i have elasticsearch cluster running. as of now there is no backup enabled till now neither S3 or NAS. we want to upgrade elasticsearch cluster on a new servers and the data size is 100gb of 2 indexes.
since we dont have backup, can we copy the data stored directory from the running cluster on all three nodes to a new cluster will this work?
current running version of es:6.2.3 to es:6.3.4
please advice.
Thanks in advance.
Taking a copy of indices folder on the running system of your elasticsearch cluster and restoring on the new es-cluster working fine.
Thanks to #Andreas Volkmann
Related
I have taken Cassandra backup from one cluster and going to restore to another(new) cluster with same configuration.
1) I have copied backup to each node in cluster (different node with different server backup )
2) Copied data into correct location in data path
But when I log into CQLSH, data does not display. Restarting the nodes also did not work.
For restoring clone cluster, you will have to export tokens from all Cassandra nodes and put them into new cassandra.yaml on each node. copy all the sstable per node after schema creation and start Cassandra services.
I manage a small ELK stack which resides on one docker host with a 3 node elasticsearch cluster, a master, a client node, and a data node each running elasticsearch 2.4.x. Each of these nodes had the same host directory bind mounted as the elasticsearch data directory even though only the data node needed it.
While testing the upgrade path to 5.x I was running into a very strange issue. The cluster would come back up, but would not initialize any of the 2.x created indices, throwing errors:
[o.e.g.DanglingIndicesState] [elastic-data] [[logstash-2017.02.01/pBco8d7dQAqmZoI37vUIOQ]] dangling index exists on local file system, but not in cluster metadata, auto import to cluster state
The indices would never initialize and remain red. The stack would create new indices fine and if I deleted these indices the system would work perfectly fine, but that data loss is definitely sub-optimal if I was going to do this on a production system.
The fact the the master and client nodes had been mounting the data directory turned out to be the cause of this issue. Elasticsearch 5.x enforces a default limit of 1 node to a data directory and while the master and client 2.x nodes had not been actively data managing, they had affected the folder structuring. I was able to get a clean upgrade with all green indices by first removing the bind mounts from master and client on the 2.x cluster and letting that sort itself out, then upgrading to 5.x. Hope this helps anybody else who runs into this issue.
We have version PHD3.0 hadoop cluster for 5 node using ambari installed on rackspace . We have to migrate into Google cloud (GCP).
But Not getting steps How to conduct following
Q1: How do i migrate data,metadata configuration from Old machine to New machine. ( Old Machine version is cantos 6.5 to target machine version centos 6.5)
Q2: What components and folders to backup? What would be the commands?
Q3: How to backup nameode and datanodes?
Q4: Do we need to take backup of ambari database also?
Any help on it would be much appreciated?
I would personally prefer to provision hadoop cluster in GCP and move data using distcp to new cluster.
For hawq managed tables move data to hdfs and then do distcp.
Some occasions on AWS i moved data to s3 and import back into hadoop.
I have a cluster of 4 elasticsearch servers and was wondering if curator will remove indices from the entire cluster when ran from one server or if I need to run it on all servers?
It deletes from the whole cluster. It uses the Elasticsearch API, rather than any file system logic.
Can I use the snapshot and restore module of elastic search to export one index to all together to a new elastic search cluster.
For e.g
Can I export all the indices from development cluster to QA cluster?
Check out https://github.com/taskrabbit/elasticsearch-dump, we made it for exactly this purpose
Yes, you can use the snapshot & restore feature to backup indices from one cluster and move them to another cluster. Once you have created the snapshot on your development cluster, you can copy it over to your QA cluster and restore it.
The advantage over just performing a direct copy of the index folder is that you do not need to shutdown the clusters. The snapshot is a point in time capture of the indices state and can be run without taking the cluster offline.