MarkLogic 9.0.8.2
We have configured MarkLogic cluster (3 nodes) as follows
PF - primary forest, RF - replica forest
Host1: PF01 RF02 RF03
Host2: PF02 RF01 RF03
Host3: PF03 RF01 RF02
In normal scenario, when we see database status page, PF should be in "open" state and and everything else will be "wait replicating" state.
Now due to some reason, state of RF01 and RF02 became in "open" state, so that means all load is now on Host3
Reindexing is in progress.
Now question is while reindexing is in progress, can we restart RF01-2 & RF02-2, so that PF01 and PF02 are again in "open" state?
Are there any best practices we should follow for forest restart?
See this knowledgbase article What Triggers Failover in MarkLogic Server
If this occurred during a reindex, I suspect that you may not have sufficient resources to support the workload. Depending on what "normal" is for your system, reindexing can cause an increase in resource demands.
Look through your ErrorLog.txt and I suspect you will see signs of trouble. Look for any log events with a severity greater than INFO (NOTICE, WARNING, ERROR, CRITICAL).
You may need to reduce the throttle of your reindexing and consider scaling options to ensure that your cluster is appropriately sized. You might consider opening a MarkLogic Support ticket, if you are eligible.
If you think that the primary hosts are ready to take over, you can "flip the forests" back to the primary by restarting the replica forests that are now open.
See the documentation: Reverting a Failed Over Forest Back to the Primary Host
Also, the following knowledgebase article provides a script that can be executed in order to flip all of the open replicas:
https://help.marklogic.com/Knowledgebase/Article/View/427/0/scripting-failover-flipping-replica-forests-back-to-their-masters-using-xquery
I believe, you should wait until your re-indexing is complete. Once the re-indexing is completed you can flip back forests. As re-indexing started when master was in open state its possible that some documents might not be synced in master forests and re-indexing completed or failed. Those documents might get missed.
Related
We have a small Elasticsearch cluster for 3 nodes: two in one datacenter and one in another for disaster recovery reasons. However, if the first two nodes fail simultaneously, the third one won't work either - it will just throw "master not discovered or elected yet".
I understand that this is intended - this is how Elasticsearch cluster should work. But is there some additional special configuration that I don't know to keep the third single node working, even if in the read-only mode?
nope, there's not. as you mentioned it's designed that way
you're probably not doing yourselves a lot of favours by running things across datacentres like that. network issues are not kind on Elasticsearch due to it's distributed nature
Elasticsearch runs in distributed mode by default. Nodes assume that there are or will be a part of the cluster, and during setup nodes try to automatically join the cluster.
If you want your Elasticsearch to be available for only node without the need to communicate with other Elasticsearch nodes. It works similar to a standalone server. To do this we can tell Elasticsearch to work in local only (disable network)
open your elasticsearch/config/elasticsearch.yml and set:
node.local: true
After follow the guide below, i manage to set up a active-passive cluster node. But i notice that when the main fails and come back on, it does not take back the primary role. What setting i should configure in the crm to ensure that when the primary recover after a fail it takes back as primary from the back up machine?
https://www.theurbanpenguin.com/drbd-pacemaker-ha-cluster-ubuntu-16-04/
You would do this via an infinity location constraint. Something like:
location fs-on-alice fs_res inf: alice
Automatic fail-back is usually not suggested. In an ideal HA configuration it shouldn't matter at all what system is currently primary. Also, I have seen instances where a node is having intermittent panics/reboots every 10-20 minutes. Now you have services failing over, and stopping/restarting, several times an hour. If not for the location constraint and automatic fail-backs you would not have this behavior.
well giving a "inf" location rule will always force the resource to run on alice only. And if alice goes down for maintenance then the resource will not run on any other node. What u want to add is called stickiness.
pcs constraint location fs_res prefers alice =50
Refer for more detail: http://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Clusters_from_Scratch/_prefer_one_node_over_another.html
The release notes of Infinispan 8 describes a new feature: Staggered remote gets.
These are described in the user guide:
11.4. Distribution Mode
The remote GET requests are staggered: we request the value from the primary owner, but if it doesn’t respond in a reasonable amount of time, we request the value from the backup owners as well.
This feature is documented for the Distribution Mode only.
Is this feature used for Replicated Mode as well?
Generally speaking: Is it safe to assume that replicated caches are a special case of distributed caches?
Generally speaking, yes it's true that Replicated Mode is a special case of Distributed Caches. The code is pretty much the same, with the exception of Replicated mode keeping an amount of replicas which is equal to the size of the cluster: each node will also be a full replica.
A Get operation will not issue a Remote Get when the current node is also an owning replica of the entry. So while it's true that a Remote Get would also be "staggered" if it the method was invoked, in practice when you have Replication you will never actually perform a Remote Get.
After going through H2 developer guide I still don't understand how can I find out what cluster node(s) was/were failing and which database needs to be recovered in the event of temporary network failure.
Let's consider the following scenario:
H2 cluster started with N active nodes (is actually it true that H2 can support N>2, i.e. more than 2 cluster nodes?)
(lots DB updates, reads...)
Network connection with one (or several) cluster nodes gets down and node becomes invisible to the rest of the cluster
(lots of DB updates, reads...)
Network link with previously disconnected node(s) restored
It is discovered that cluster node was probably missing (as far as I can see SELECT VALUE FROM INFORMATION_SCHEMA.SETTINGS WHERE NAME='CLUSTER' starts responding with empty string if one node in cluster fails)
After this point it is unclear how to find out what nodes were failing?
Obviously, I can do some basic check like comparing DB size, but it is unreliable.
What is the recommended procedure to find out what node was missing in the cluster, esp. if query above responds with empty string?
Another question - why urlTarget doesn't support multiple parameters?
How I am supposed to use CreateCluster tool if multiple nodes in the cluster failed and I want to recover more than one?
Also I don't understand how CreateCluster works if I had to stop the cluster and I don't want to actually recover any nodes? What's not clear to me is what I need to pass to CreateCluster tool if I don't actually need to copy database.
That is partially right SELECT VALUE FROM INFORMATION_SCHEMA.SETTINGS WHERE NAME='CLUSTER', will return an empty string when queried in standard mode.
However, you can get the list of servers by using Connection.getClientInfo() as well, but it is a two-step process. Paraphrased from h2database.com:
The list of properties returned by getClientInfo() includes a numServers property that returns the number of servers that are in the connection list. getClientInfo() also has properties server0..serverN, where N is the number of servers - 1. So to get the 2nd server from the list you use getClientInfo('server1').
Note: The serverX property only returns IP addresses and ports and not
hostnames.
And before you say simple replication, yes that is default operation, but you can do more advanced things that are outside the scope of your question in clustered H2.
Here's the quote for what you're talking about:
Clustering can only be used in the server mode (the embedded mode does not support clustering). The cluster can be re-created using the CreateCluster tool without stopping the remaining server. Applications that are still connected are automatically disconnected, however when appending ;AUTO_RECONNECT=TRUE, they will recover from that.
So yes if the cluster stops, auto_reconnect is not enabled, and you stick with the basic query, you are stuck and it is difficult to find information. While most people will tell you to look through the API and or manual, they haven't had to look through this one so, my sympathies.
I find it way more useful to track through the error codes, because you get a real good idea of what you can do when you see how the failure is planned for ... here you go.
Let's say I have 3 nodes. 1 of which is the master.
I have an API (running on another machine) which hits the master and gets my search result. This is through a subdomain, say s1.mydomain.com:9200 (assume the others are pointed to by s2.mydomain.com and s3.mydomain.com).
Now my master fails for whatever reason. How would my API recover from such a situation? Can I hit either S2 or S3 instead? How can I figure out what the new master is? Is there a predictable way to know which one would be picked as the new master should the master go down?
I've googled this and it's given me enough information about how when a master goes down, a failover is picked as the new master but I haven't seen anything clarify how I would need to handle this from the outside looking in.
The master in ElasticSearch is really only for internal coordination. There are no actions required when a node goes down, other than trying to get it back up to get your full cluster performance back.
You can read/write to any of the remaining nodes and the data replication will keep going. When the old master node comes back up, it will re-join the cluster once it has received the updated data. In fact, you never need to worry if the node you are writing on is the master node.
There are some advanced configurations to alter these behaviors, but ElasticSearch comes with suitable defaults.