I have cross-datacenter replication (XDCR) setup to replicate data from a Couchbase bucket to an Elasticsearch index. Some times, however, Elasticsearch fails and needs to be restarted. During this time any changes to the Couchbase bucket are not replicated. Is there a recommended way of dealing with this? Ideally, once Elasticsearch restarts the data added to Couchbase in the interim gets replcated successfully.
Related
I am using EBS for storage for my Elasticsearch cluster on EKS cluster. But in terms of performance, I want to use instance storage instead of EBS. In case the critical point instance closes, shard or replicas will be lost. How can I go about the configuration properly without losing in this scenario?
I've made sample changes to a few parameters I created for storage in the configuration files, but I'm not sure it's the right way. I left it as is so that the changes I made do not cause any data loss.
I am working on an Elasticsearch indexing task.
Currently we are indexing hundreds of thousands of documents to ES cluster (many ES instances) on daily basis. We simply read data from DB and various of data sources, collate and compile them and directly index them to ES using python elasticsearch-dsl lib.
Currently;
Now we want use Logstash between the application server and ES however we don't want to change our current codebase. Thus we could easily switch between ES to Logstash and keep business logic in our application.
My question is how can I use logstash as a transparent middleware. I simply want logstash to forward all rest messages as they are to the ES cluster.
Want to;
I have a couchbase cluster setup as the primary source for data. From this a subset of data is synced to a elasticsearch cluster via the Couchbase Transport Plugin for ElasticSearch(https://github.com/couchbaselabs/elasticsearch-transport-couchbase) which sets up an XDCR stream from couchbase to elasticsearch.
Due to some issues with the elasticsearch cluster all data needs to be synced again from couchbase to elasticsearch. I have tried recreating XDCR but that does not seem to help as it only copies a very small subset of documents. Is there a way by which this can be achieved?
Additional details
Couchbase version: 3.1.0
Number of couchbase documents: 50K+
Documents synced to elasticsearch: around 700 (expected 20K+)
If a document in couchbase is modified it is successfully synced to elasticsearch
The issue you're experiencing is likely in one of the following: XDCR, the Couchbase Transport Plugin for Elasticsearch, or Elasticsearch itself.
Start by checking for XDCR errors. You can find your XDCR logs using these instructions. Be aware that the Transport Plugin uses XDCR v1 and almost everything else in Couchbase uses v2.
Consult the advice in troubleshooting the Couchbase Transport Plugin for Elasticsearch. Instructions should work for you even though they are from the 4.0 docs.
Pay attention to how your documents are being mapped to Elasticsearch. You mention that you're expecting only a subset of documents to be synced to Elasticsearch, so it's possible that you have lost a setting or misconfigured something. You can enable logging and observe a small set of test data. At TRACE level, you should be able to see each document that is inspected.
If all of that fails, make sure the basics are working by indexing the beer sample dataset, following the directions in the Couchbase docs. ES is probably not the issue, but test with a fresh ES instance will rule out problems on that side.
I'm having an issue with my Elasticsearch replication of my Couchbase DB.
Certain documents in my DB are not being replicated correctly. Rather than replicating as full couchbase documents, there is an entry with _type of couchbaseCheckpoint pointing to certain documents. These checkpoints never seem to update or grab the documents correctly, even after a refresh. This seems to occur at random, with some documents replicating correctly and being stored as full couchbaseDocuments, while others stay as these checkpoints. I'm not seeing any errors related to this in my logs at all.
versions:
couchbase 3.0.1
elasticsearch 1.3
couchbase elasticsearch plugin 2.0.0
{
"name":"xxx",
"age": "1",
}
I have lot of data indexed in my elasticsearch.
I deleted elasticsearch folder and then extarct again fresh zip of elasticsearch and start the elasticsearch server.
I am surprised because after staring new elasticsearch server, I again found all old data and this problem persists again and again.
Can any please help me? I don't want to get all old data indexed in elasticsearch.
Regards
Given the cluster health response it's not a problem with multiple nodes running on the same cluster as suggested by Igor. I'd suggest you to check the java processes running. You could maybe have an elasticsearch hanging somewhere which keeps writing in that folder.