we have a HA deployment of mongodb including multiple shards and 2 mongos nodes. all access to the underlying mongod instances is done via these mongos nodes.
my question is if there is a possibility to configure these 2 mongos instances so spring-data-mongodb will either load balance between them or at least try to connect the second one i the first is not available
Thanks,
Uzi
Related
There are lot of tutorials how to use Hazelcast with spring boot in kubernetes for example
Problem is we have 2 data centers, A and B. In both there of these data centers there is kubernetes cluster with application and hazelcast cluster. So the question is how should I configure configure application in data center A to save data in both cluster (A is easy according to tutorial) but how to also configure hazelcast in second cluster ?
You can solve this problem on two layers:
1. Application layer - Your application can store the data in multiple Hazelcast clusters. If you use Hazelcast with Spring Data, then you can configure multiple databases with Spring JPA. Note however that when your application stores the data in the data centre that is in a different region, then the latency will suffer significantly
2. Hazelcast layer - Hazelcast clusters propagate the data changes across multiple data centres using geo-replication (Hazelcast WAN Replication, Enterprise feature); then you connect your application to on Hazelcast cluster only, the one that is co-located with your application.
I have a application using Redis. This system implemented with java spring used jedis package for connection to the redis with the configuration as follow
jedis.pool.host=redisServer-IP
so the application connect to redis server on the redisServer-IP and works fine but, for the lack of memory on a single server and and HA capability I need to use a redis cluster I used docker compose to create a redis cluster using the here.
Also redis cluster working fine with three masters and three replicas.
I just need to understand, the Redis Cluster can work with the single endpoint, because I can only set single endpoint in the above jedis.pool.host configuration, or I need to have a proxy to deal with the redis cluster ?
NOTE: I can not make any changes in my application
We created redid cluster with redis 5. In our cluster we have 3 master and 6 slave nodes which is recommended by Redis.
Now to connect redid cluster via spring boot, we used
spring.redis.cluster.nodes[0]=192.168.2.4:7379
.
.
This way we configured all redis nodes in spring boot configuration including 3 masters and 6 slave. Want to understand, is it required to pass all redis node configuration to spring boot? Can we just go ahead with master node configurations in spring boot? If possible, please share reason behind the suggestion.
I am a newbie to Kafka technology.
I have setup a basic single node cluster using Ambari.
I want to understand what is the recommended configuration for a production server. Let's say in production I will have 5 topics each getting traffic in the range of 500,000 to 50,000,000 in a day.
I am thinking of setting up a 3-4 node kafka cluster using EC2 r5.xlarge instances.
I am mostly confused about zookeeper part. I understand zookeeper needs odd number of nodes and zookeeper is installed on all kafka nodes, then how do I run Kafka with even number of nodes. If this is true it will limit Kafka to odd number of nodes as well.
Is it really needed to install Zookeeper on all Kafka nodes. Can I install Zookeeper on separate nodes and Kafka brokers on separate nodes, how ?
What if I want to run multiple Kafka clusters. Is it possible to manage multiple Kafka clusters through single Zookeeper cluster, how if possible ?
I have started learning Kafka recently only, any help would be appreciated.
Thanks,
I am mostly confused about zookeeper part. I understand zookeeper
needs odd number of nodes and zookeeper is installed on all kafka
nodes, then how do I run Kafka with even number of nodes. If this is
true it will limit Kafka to odd number of nodes as well.
Zookeeper can, but doesn't have to be installed on the same servers as kafka. It is not requirement to run zookeeper on odd number of nodes, just very good recommendation
Is it really needed to install Zookeeper on all Kafka nodes. Can I
install Zookeeper on separate nodes and Kafka brokers on separate
nodes, how ?
It is not required and it's even better not to have zookeeper and kafka on the same server. Installing zookeeper on another server is quite similar to when they reside on the same one. Every kafka broker needs to have zookeeper.connect setting pointing to all zookeeper nodes.
What if I want to run multiple Kafka clusters. Is it possible to
manage multiple Kafka clusters through single Zookeeper cluster, how
if possible ?
It is possible. In this case it's recommended to have servers dedicated just to zookeeper ensemble. In this case, in zookeeper.connect settings you should use hostname:port/path instead just hostname:port.
Can I install Zookeeper on separate nodes and Kafka brokers on separate nodes, how ?
You can, and you should if you have the available resources.
Run zookeeper-server-start zookeeper.properties on an odd number of servers. (max 5 or 7 for larger Kafka clusters)
On every other machine that is a Kafka broker, not the same servers as Zookeeper, edit server.properties to point to that set of Zookeeper machine addresses for the zookeeeper.connect property.
Then do kafka-server-start server.properties for every new Kafka broker.
From there, you can scale Kafka independently of Zookeeper
Is it possible to manage multiple Kafka clusters through single Zookeeper cluster
Look up Zookeeper chroots
One Kafka cluster would be defined as
zoo1:2181/kafka1
And a second
zoo1:2181/kafka2
be careful not to mix those up if machines shouldn't be in the same Kafka cluster
You can find various CloudFormation, Terraform, or Ansible repos for setting up Kafka in a distibuted way in the Cloud on Github, or go for Kubernetes if you are familiar with it.
I want to create a distributed cluster in spring xd.
I am able to create a cluster with single admin, one zookeeper, one instance of redis and hsqldb.
But when i'm trying to do that with multiple instance of zookeeper , hsqldb, redis ,i'm not able to configure it correctly.
You should only have a single instance of zookeeper, hsqldb and redis. All xd-admins should be configured to connect to the same instance of each of these services and so should the xd-containers be.
Like Thomas has mentioned, the idea is that you have your (multiple) instances of admin and containers deployed, and all connect to the same zk,redis, hsqldb & rabbitmq.
Why do you want to start multiple instances of these applications?
Zookeeper provides the topology of the cluster and manages deployments. Also, it makes sure to note when nodes go up and down - avoiding single point of failures when you have many xd-admin instances (one is leader and the others replicate, they will become leader if the current one fails).
Or are you talking about making those instance parallel to avoid a SPOF? In that case, you should try to dedicate an entire VM for each of those applications.