How does Multiple shards per module support works in ODL nitrogen - opendaylight

The module-shards.conf file mentions the following :
For Helium we support only one shard per module. Beyond Helium we will support more than 1
Have ODL Nitrogen and trying to use "module-shards.conf" like the following :
module-shards = [
{
name = "default"
shards = [
{
name="default1"
replicas = [
"member-1"
]
},
{
name="default2"
replicas = [
"member-1"
]
}
]
}
]
That seems to be working as karaf shows the 2 shards successfully created
for the same module "default". But how the data is distributed among
the 2 shards is not clear, can application control/decide which data goes in which of the module shards.
I am not clear how the application config tree data and the operational tree
data for a given module be stored under different shards for the same module.
If there are multiple shards for the same module, can application decide/control
which shard to use for storing a particular type of data. For example,
can application decide to store the config tree in "default1" shard
and operational tree in "default2" shard for the same module "default" ?
Is it possible to disable operational tree component for one of the shards of
a given module, say disable operational tree component for shard "default2"
of module "default" with 2 shards namely "default1" and "default2" ?

While it says for Helium only one shard per module is supported, that is still the case - support for more than 1 was never implemented. There hasn't been a use case for that. The shards are created but transactions are only routed to one of them (ie picks the first one that is found).

Related

Performance when using NFS storage for Elasticsearch

I have a server with 32 cores, 62 GB of RAM but we have NFS storage and I think it's starting to bottleneck our daily work. In our Kibana errors like queue_size are appearing more frequently. We just got a new (same) server to use it as a replica and share the load, will this help? What other recomendations you have? We have multiple dashboards with like 20 different variables each, will they be evenly distributed between the primary node and the replica? Unfortunately, local storage is not an option.
Are you actively indexing data on these nodes? If yes you can increase refresh_interval
PUT /myindex/_settings
{
"index" : {
"refresh_interval" : "30s"
}
}
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html to make system less demanding for IO. You can completely disable refresh functionality and trigger it manually.
PUT /myindex/_settings
{
"index" : {
"refresh_interval" : "-1"
}
}
POST /myindex/_refresh
Take a look on Bulk API it significantly decrease load on indexing stage.
Adding new servers to cluster helps too. Elasticsearch designed to scale horizontally. From my experience you can run 6-8 virtual nodes on server you have described. Put more shards to evenly distribute load.
Do you see what is your bottleneck (Lan, IO, CPU)?

Creating cluster member removes some configuration

I'm using WAS ND and want to have dmgr profile with federated managed profile app.
I am creating cluster using:
AdminTask.createCluster('[-clusterConfig [-clusterName %s -preferLocal true]]' % nameOfModulesCluster)
Next, I'm configuring my WAS instance, queues, datasources, jdbc, JMS Activation Specs, factories etc.
By the time I want to create cluster member, I'm displaying:
print("QUEUES: \n" + AdminTask.listSIBJMSQueues(AdminConfig.getid('/ServerCluster:ModulesCluster/')))
print("JMS AS: \n" + AdminTask.listSIBJMSActivationSpecs(AdminConfig.getid('/ServerCluster:ModulesCluster/')))
And it returns all queues I've created earlier. But when I'm calling
AdminTask.createClusterMember('[-clusterName %(cluster)s -memberConfig [-memberNode %(node)s -memberName %(server)s -memberWeight 2 -genUniquePorts true -replicatorEntry false] -firstMember [-templateName default -nodeGroup DefaultNodeGroup -coreGroup DefaultCoreGroup -resourcesScope cluster]]' % {'cluster': nameOfCluster, 'node': nameOfNode, 'server': nameOfServer})
AdminConfig.save()
configuration displayed earlier is... gone. Some configuration (like datasources) is still displayed in ibm/console, but queues and jms as are not. The same print is displaying nothing, but member is added to cluster.
I can't find any information using google. I've tried AdminNodeManagement.syncActiveNodes(), but it won't work since I'm using
/opt/IBM/WebSphere/AppServer/bin/wsadmin.sh -lang jython -conntype NONE -f global.py
and AdminControl is not available.
What should I do in order to keep my configuration created before clustering? Do I have to sync it somehow?
This is the default behavior and is due to the -resourcesScope attribute in the createClusterMember command. This attribute determines how the server resources are promoted in the cluster, while adding the first cluster member.
Valid options for resourcesScope are :
Cluster: moves the resources of the first cluster member to the cluster level. The resources of the first cluster member replace the resources of the cluster. (is the default option)
Server: maintains the server resources at the new cluster member level. The cluster resources remain unchanged.
Both: copies the resources of the cluster member (server) to the cluster level. The resources of the first cluster member replace the resources of the cluster. The same resources exist at both the cluster and cluster member scopes.
Since you have set "-resourcesScope cluster" in your createClusterMember command, all configuration created at cluster scope are being deleted/replaced by the empty configurations of the new cluster member.
So, for your configurations to work, set "-resourcesScope server", such that the cluster configurations are not replaced by the cluster member configurations.
AdminTask.createClusterMember('[-clusterName %(cluster)s -memberConfig [-memberNode %(node)s -memberName %(server)s -memberWeight 2 -genUniquePorts true -replicatorEntry false] -firstMember [-templateName default -nodeGroup DefaultNodeGroup -coreGroup DefaultCoreGroup -resourcesScope server]]' % {'cluster': nameOfCluster, 'node': nameOfNode, 'server': nameOfServer})
AdminConfig.save()
Refer "Select how the server resources are promoted in the cluster" section in https://www.ibm.com/support/knowledgecenter/en/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/urun_rwlm_cluster_create2_v61.html for more details.

es_rejected_execution_exception rejected execution

I'm getting the following error when doing indexing.
es_rejected_execution_exception rejected execution of org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1#16248886
on EsThreadPoolExecutor[bulk, queue capacity = 50,
org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor#739e3764[Running,
pool size = 16, active threads = 16, queued tasks = 51, completed
tasks = 407667]
My current setup:
Two nodes. One is the master (data: true, master: true) while the other one is data only (data: true, master: false). They are both EC2 I2.4XL (16 Cores, 122GB RAM, 320GB instance storage). 2 shards, 1 replication.
Those two nodes are being fed by our aggregation server which has 20 separate workers. Each worker makes bulk indexing request to our ES cluster with 50 items to index. Each item is between 1000-4000 characters.
Current server setup: 4x client facing servers -> aggregation server -> ElasticSearch.
Now the issue is this error only started occurring when we introduced the second node. Before when we had one machine, we got consistent indexing throughput of 20k request per second. Now with two machine, once it hits the 10k mark (~20% CPU usage)
we start getting some of the errors outlined above.
But here is the interesting thing which I have noticed. We have a mock item generator which generates a random document to be indexed. Generally these documents are of the same size, but have random parameters. We use this to do the stress test and check the stability. This mock item generator sends requests to aggregation server which in turn passes them to Elasticsearch. The interesting thing is, we are able to index around 40-45k (# ~80% CPU usage) items per second without getting this error. So it seems really interesting as to why we get this error. Has anyone seen this error or know what could be causing it?

percolate returns empty matches under heavy load during elasticsearch cluster resizing

We have an elasticsearch cluster dynamically re-sizing in respect to percolate message count in a rabbitmq queue.
We have a single shard and ~18K query in our index, and we use auto_expand_replicas: "0-all" at index settings to copy single shard to all nodes when a node becomes available.
But during heavy load and cluster re-sizing, some requests produces unexpected empty matches.
We send ~1M percolate request daily and we were losing ~1K content. We added a cluster status control to our code, if cluster status is not green before and after percolate request we're waiting for green status and re-sending percolate request, we were able to reduce lost content count from 1K to ~100 in this way. We do not live this problem in a cluster with fixed node size.
Unfortunately any loss is not acceptable in our scenario, and we don't want to give up auto scaling, we need to find a workaround for this problem.
To repeat problem, you can use following bash script:
https://gist.github.com/ekesken/de41598a1e7e54c6f33c
This script will download and install elasticsearch 1.5.2 on your current directory, create a cluster with 10 nodes on your local and create index and percolation queries and will start testing.
Normally we expect following output for single percolate request:
curl -XGET 'localhost:9200/my-index/my-type/_percolate' -d '{
"doc" : {
"message" : "A new bonsai tree in the office"
}
}'
{"took":95,"_shards":{"total":1,"successful":1,"failed":0},"total":1,"matches":[{"_index":"my-index","_id":"tree"}]}
After running script, if you see all shards in all nodes are started at http://localhost:9200/_cat/shards response and test script is still running, that means you couldn't reproduce problem, try increasing node count which was 10 by default:
./repeat_percolation_loss.sh 15 test-only
When you reproduce problem, script will exit with following output:
{"took":209,"_shards":{"total":1,"successful":1,"failed":0},"total":0,"matches":[]}
Problem repeated! Congratulations.
You can shutdown all servers and clean all directory and files created via script with command:
./repeat_percolation_loss.sh 15 clean
Change node count above with latest node count you've tried.

How to check vnode disabled on a hadoop node

Linking back to this question:
Why not enable virtual node in an Hadoop node?
I'm running a mixed 3 node cluster with 2 cassandra and 1 analytics nodes and disabled the virtual nodes by generating 3 tokens with the utility given by DataStax enterprise.
But when I run 'nodetool status' command, I still see 256 tokens with each node and when a mapreduce job is created, it creates 257 mappers and takes a very long time to execute a query with small data.
So my specific questions are:
Is virtual node setting still not disabled? How can I verify if its disabled?
If its disabled then why 257 mappers are still created for each job? Is there a different configuration for that?
Thanks much for any help!!
1) It's not disabled. You can tell because it still says 256 tokens in nodetool status.
To disable vnodes you need to make sure that you change the num_tokens variable in the cassandra.yamnl
# If you already have a cluster with 1 token per node, and wish to migrate to
# multiple tokens per node, see http://wiki.apache.org/cassandra/Operations
# num_tokens: 256 << Make sure this line is commented out
# initial_token allows you to specify tokens manually. While you can use it with
# vnodes (num_tokens > 1, above) -- in which case you should provide a
# comma-separated list -- it's primarily used when adding nodes to legacy clusters
# that do not have vnodes enabled.
initial_token: << Your generated token goes here

Resources