I have 3 redis master nodes and two slave nodes for each master. How do I write configuration for those masters and slaves in springboot?
Related
I am currently using one Hadoop cluster of 10 nodes (1 Name Node and 9 Data Nodes) in which Hbase, Hive, Kafka, Zookeeper and other echo systems of Hadoop are running. Now I want to fetch data from RDBMS and store it in HDFS in real time. Can we do that by using Confluent Source Connector and HDFS2 Sink Connector with in the same cluster or do I need to have a separate cluster for Kafka Connect?
Yes. Kafka Connect is a standalone Java process, just like each of the other components you mentioned.
do I need to have a separate cluster for Kafka Connect
That would be preferred, but is optional
We created redid cluster with redis 5. In our cluster we have 3 master and 6 slave nodes which is recommended by Redis.
Now to connect redid cluster via spring boot, we used
spring.redis.cluster.nodes[0]=192.168.2.4:7379
.
.
This way we configured all redis nodes in spring boot configuration including 3 masters and 6 slave. Want to understand, is it required to pass all redis node configuration to spring boot? Can we just go ahead with master node configurations in spring boot? If possible, please share reason behind the suggestion.
I am a newbie to Kafka technology.
I have setup a basic single node cluster using Ambari.
I want to understand what is the recommended configuration for a production server. Let's say in production I will have 5 topics each getting traffic in the range of 500,000 to 50,000,000 in a day.
I am thinking of setting up a 3-4 node kafka cluster using EC2 r5.xlarge instances.
I am mostly confused about zookeeper part. I understand zookeeper needs odd number of nodes and zookeeper is installed on all kafka nodes, then how do I run Kafka with even number of nodes. If this is true it will limit Kafka to odd number of nodes as well.
Is it really needed to install Zookeeper on all Kafka nodes. Can I install Zookeeper on separate nodes and Kafka brokers on separate nodes, how ?
What if I want to run multiple Kafka clusters. Is it possible to manage multiple Kafka clusters through single Zookeeper cluster, how if possible ?
I have started learning Kafka recently only, any help would be appreciated.
Thanks,
I am mostly confused about zookeeper part. I understand zookeeper
needs odd number of nodes and zookeeper is installed on all kafka
nodes, then how do I run Kafka with even number of nodes. If this is
true it will limit Kafka to odd number of nodes as well.
Zookeeper can, but doesn't have to be installed on the same servers as kafka. It is not requirement to run zookeeper on odd number of nodes, just very good recommendation
Is it really needed to install Zookeeper on all Kafka nodes. Can I
install Zookeeper on separate nodes and Kafka brokers on separate
nodes, how ?
It is not required and it's even better not to have zookeeper and kafka on the same server. Installing zookeeper on another server is quite similar to when they reside on the same one. Every kafka broker needs to have zookeeper.connect setting pointing to all zookeeper nodes.
What if I want to run multiple Kafka clusters. Is it possible to
manage multiple Kafka clusters through single Zookeeper cluster, how
if possible ?
It is possible. In this case it's recommended to have servers dedicated just to zookeeper ensemble. In this case, in zookeeper.connect settings you should use hostname:port/path instead just hostname:port.
Can I install Zookeeper on separate nodes and Kafka brokers on separate nodes, how ?
You can, and you should if you have the available resources.
Run zookeeper-server-start zookeeper.properties on an odd number of servers. (max 5 or 7 for larger Kafka clusters)
On every other machine that is a Kafka broker, not the same servers as Zookeeper, edit server.properties to point to that set of Zookeeper machine addresses for the zookeeeper.connect property.
Then do kafka-server-start server.properties for every new Kafka broker.
From there, you can scale Kafka independently of Zookeeper
Is it possible to manage multiple Kafka clusters through single Zookeeper cluster
Look up Zookeeper chroots
One Kafka cluster would be defined as
zoo1:2181/kafka1
And a second
zoo1:2181/kafka2
be careful not to mix those up if machines shouldn't be in the same Kafka cluster
You can find various CloudFormation, Terraform, or Ansible repos for setting up Kafka in a distibuted way in the Cloud on Github, or go for Kubernetes if you are familiar with it.
I am creating a spring boot microservices application with 3 microservices.
I'm using consul for discovery. I have only a single data centre with 4 hosts.
Should I have 3 microservices connecting to 3 consul agents(clients) in their respective localhosts and a consul server running on the remaining one host?
It is mentioned that a cluster should have more than one server. Is it applicable in above situation?
Your services should connect to an agent on localhost.
Consul servers should be run in a cluster. A 3 node cluster can handle 1 node failure, a 5 node cluster can handle 2 nodes failing. See https://www.consul.io/docs/internals/consensus.html#deployment_table
Yes, the cluster is recommended to have more than 1 server and the microservices can connect to an agent on localhost.
we have a HA deployment of mongodb including multiple shards and 2 mongos nodes. all access to the underlying mongod instances is done via these mongos nodes.
my question is if there is a possibility to configure these 2 mongos instances so spring-data-mongodb will either load balance between them or at least try to connect the second one i the first is not available
Thanks,
Uzi