Current IBM MQSeries Queue Depth in our System is 150000.Because of Huge queue depth, IBM MQ Cache is getting Corrupted/Going down. Is there Something we can do to have Queue Manager hold current queue depth of 150000 given that IBM Recommended Queue depth is 5000?
Related
We have 3 node of kafka cluster with around 32 topic and 400+ partition
spread across these servers. We have the load evenly distributed amongst
this partition however we are observing that 2 broker server are running
around >60% CPU where as the third one is running just abour 10%. How do we
ensure that all server are running smoothly? Do i need to reassing the
partition (kafka-reassign-parition cmd).
PS: The partition are evenly distributed across all the broker servers.
In some cases, this is a result of the way that individual consumer groups determine which partition to use within the __consumer_offsets topic.
On a high level, each consumer group updates only one partition within this topic. This often results in a __consumer_offsets topic with a highly uneven distribution of message rates.
It may be the case that:
You have a couple very heavy consumer groups, meaning they need to update the __consumer_offsets topic frequently. One of these groups uses a partition that has the 2nd broker as its leader. The other uses a partition that has the 3rd broker as its leader.
This would result in a significant amount of the CPU being utilized for updating this topic, and would only occur on the 2nd and 3rd brokers (as seen in your screenshot).
A detailed blog post is found here
I have a kafka cluster with three brokers and one topic with replication factor of three and three partitions. I can see that every broker has a copy of log for all partitions with the same size. There are two producers for this topic.
One day I reduced writing volume of one producer by half. Then I found that all three brokers' inbound traffic reduced which is expected, but only partition 1's leader node's out traffic reduced which I don't understand.
The partition leader's outbound traffic reduced because of replication. But each broker is the leader of one partition, why only one leader's outbound traffic reduced? Is it possible that the producer only writes content to one partition? while I don't think so.
Please help me explain it. The cluster is working fine now, but I need to understand it in case of potential problem.
Assuming you are using Default Partitioner for KafkaProducer, which means two events with the same key are guaranteed to be sent to the same partition.
From From Kafka Documentation
All reads and writes go to the leader of the partition and Followers
consume messages from the leader just as a normal Kafka consumer would
and apply them to their own log.
You could have reduced data ( from a producer) by skiping specific key or set of Keys, which could means no data to particular partition.
This answers why leader's outbound traffic reduced (No records for followers to consume)
Consider there are 3 top level queues, q1,q2,q3.Capacity Scheduler
Users of q1 and q2 submit their jobs to their respective queues, they are guaranteed to get their allocated resources. Now the resources which are not utilized by q3 has to be utilized by q1 and q2. What factors does yarn consider while dividing the extra resources? Who (q1,q2) gets preference?
Every queue in the Capacity Scheduler has 2 important properties (which are defined in terms of percentage of total resources available), which determine the scheduling:
Guaranteed capacity of the queue (determined by configuration "yarn.scheduler.capacity.{queue-path}.capacity")
Maximum capacity to which the queue can grow (determined by the configuration "yarn.scheduler.capacity.{queue-path}.maximum-capacity"). This puts an upper limit on the resource utilization by a queue. The queue cannot grow beyond this limit.
The Capacity Scheduler, organizes the queues in a hierarchical fashion.
Queues are of 2 types "parent" and "leaf" queues. The jobs can only be submitted to the leaf queues.
"ROOT" queue is the parent of all the other queues.
Each parent queue, sorts the child queues based on the demand (what's the current used capacity of the queue? Whether it is under-served or over-served?).
For each queue, the ratio (Used Capacity / Total Cluster Capacity) gives an indication about the utilization of the queue. The parent queue always gives priority to the most under-served child queue..
When the free resources are given to a parent queue, the resources are recursively distributed to the child queues, depending on the current used capacity of the queue.
Within a leaf queue, the distribution of capacity can happen based on certain user limits (for e.g. configuration parameter: yarn.scheduler.capacity.{queue-path}.minimum-user-limit-percent, determines the minimum queue capacity that each user is guaranteed to have).
In your example, for the sake of simplicity, let's assume that the queues q1, q2 and q3 are directly present under "ROOT". As mentioned earlier, the parent queue keeps the queues sorted based on their utilization.
Since q3 is not utilized at all, the parent can distribute the un-utilized resources of q3, between q1 and q2.
The available resources are distributed based on following factors:
If both q1 and q2 have enough resources to continue scheduling their jobs, then there is no need to distribute the available resources from q3
If both q1 and q2 have hit maximum capacity ("yarn.scheduler.capacity.{queue-path}.maximum-capacity", this configuration limits the elasticity of the queues. Queues cannot demand more than the percentage configured by this parameter), then the free resources are not allotted
If any one of the queues q1 or q2 is under-served, then the free resources are allotted to the under-served queue
If both q1 and q2 are under-served, then the most under-served queue is given the top priority.
The Capacity Scheduler allows sharing of Hadoop cluster along organizational lines, whereby each organization is allocated a certain capacity of the overall cluster.
I want to know that if large data come, then the capacity allocated to that to certain queue will be change automatically?
in capacity scheduler config we define yarn.scheduler.capacity.root.<queue name>.capacity and yarn.scheduler.capacity.root.<queue name>.maximum-capacity
yarn.scheduler.capacity.root.<queue name>.capacity is the capacity of queue while yarn.scheduler.capacity.root.<queue name>.maximum-capacity is maximum resources all jobs/users in that queue can take
if large data come, then the capacity allocated to that to certain queue will be change automatically.
No, queue size is fixed and doesn't change automatically according to input data volume.
you can manually change it in capacity-scheduler.xml and then refresh queues by yarn rmadmin -refreshQueues
you can write a script which will update (and refresh) the queues capacity according to input data volume but I don't think it is recommended.
What is difference between partition and replica of a topic in kafka cluster.
I mean both store the copies of messages in a topic. Then what is the real diffrence?
When you add the message to the topic, you call send(KeyedMessage message) method of the producer API. This means that your message contains key and value. When you create a topic, you specify the number of partitions you want it to have. When you call "send" method for this topic, the data would be sent to only ONE specific partition based on the hash value of your key (by default). Each partition may have a replica, which means that both partitions and its replicas store the same data. The limitation is that both your producer and consumer work only with the main replica and its copies are used only for redundancy.
Refer to the documentation: http://kafka.apache.org/documentation.html#producerapi
And a basic training: http://www.slideshare.net/miguno/apache-kafka-08-basic-training-verisign
Topics are partitioned across multiple nodes so a topic can grow beyond the limits of a node. Partitions are replicated for fault tolerance. Replication and leader takeover is one of the biggest difference between Kafka and other brokers/Flume. From the Apache Kafka site:
Each partition has one server which acts as the "leader" and zero or
more servers which act as "followers". The leader handles all read and
write requests for the partition while the followers passively
replicate the leader. If the leader fails, one of the followers will
automatically become the new leader. Each server acts as a leader for
some of its partitions and a follower for others so load is well
balanced within the cluster.
partition: each topic can be splitted up into partitions for load balancing (you could write into different partitions at the same time) & scalability (the topic can scale up without the instance limitations); within the same partition the records are ordered;
replica: for fault-tolerant durability mainly;
Quotes:
The partitions of the log are distributed over the servers in the Kafka cluster with each server handling data and requests for a share of the partitions. Each partition is replicated across a configurable number of servers for fault tolerance.
There is a quite intuitive tutorial to explain some fundamental concepts in Kafka: https://www.tutorialspoint.com/apache_kafka/apache_kafka_fundamentals.htm
Furthermore, there is a workflow to get you through the confusing jumgle: https://www.tutorialspoint.com/apache_kafka/apache_kafka_workflow.htm
Partitions
A topic consists of a bunch of buckets. Each such bucket is called a partition.
When you want to publish an item, Kafka takes its hash, and appends it into the appropriate bucket.
Replication Factor
This is the number of copies of topic-data you want replicated across the network.
In simple terms, partition is used for scalability and replication is for availability.
Kafka topics are divided into a number of partitions. Any record written to a particular topic goes to particular partition. Each record is assigned and identified by an unique offset. Replication is implemented at partition level. The redundant unit of topic partition is called replica. The logic that decides partition for a message is configurable. Partition helps in reading/writing data in parallel by splitting in different partitions spread over multiple brokers. Each replica has one server acting as leader and others as followers. Leader handles the read/write while followers replicate the data. In case leader fails, any one of the followers is elected as the leader.
Hope this explains!
Further Reading
Partitions store different data of the same type and
Yes, you can store the same message in different topic partitions but your consumers need to handle duplicated messages.
Replicas are a copy of these partitions in other servers.
Your number of replicas will be defined by the number of kafka brokers (servers) of your cluster
Example:
Let's suppose you have a Kafka cluster of 3 brokers and inside you have a topic with name AIRPORT_ARRIVALS that receives messages of Flight information and it has 3 partitions; partition 1 for flight arrivals from airline A, partition 2 from airline B, and partition 3 from airline C. All these messages will be initially written in one broker (leader) and a copy of each message will be stored/replicated to the other 2 Kafka broker (followers). Disclaimer; this example is only for an easier explanation and not an ideal way to define a message key because you could ending up with unbalanced load over specific partitions.
Partitions are the way that Kafka provides redundancy.
Kafka keeps more than one copy of the same partition across multiple brokers.
This redundant copy is called a Replica. If a broker fails, Kafka can still serve consumers with the replicas of partitions that failed broker owned