I'm only getting about 6 mb/s throughput for binary messages on one connection over the loopback. The actual messages are about 100k to 1mb in size. Any suggestions on how to debug this?
Related
We are running production single server RabbitMQ (3.7) where around 500 mobile applications are connected as producers (MQTT) and around 10 server applications as consumers. Those 500 publishers push messages basically into one queue and less often in the another one.
Recently we had issue with spikes of stacked messages in all our queues. Numbers of stacked messages went from 1 to 1000. This spike was caused by decrease of consumer rate.
I tired to find what happened and how to eliminate spikes in queues and I should limit queue length or eliminate connections. But we can’t limit we have to perform better. I took a look into RabbitMQ memory usage, cpu and same for consumers everything looks fine and RabbitMq was running around 50% on total load same for memory. Also consumers doesn’t seems to be a bottleneck because consume rate went event higher after the queue length grown.
I have a couple of questions:
Is RabbitMQ designed for such a large amount of consumers?
I read that each queue is single threaded is it possible that rabbit just can’t handle 500 producers in one queue and throughput gets lower?
What else I can use to tackle the cause of lower consumer rate? Number of threads in Rabbit?
What do you recon to measure or test benchmark/performance of RabbitMQ server?
We are writing messages at the rate of about 9000 records/sec into our kafka cluster, at times we see that the producer performance degrades considerably and then it never recovers. When this happens we see the following error "unable to allocate buffer within timeout". Below are the JMX producer metrics taken when the process is running well and when it reaches the bad state. The "waiting-threads" metric is very high when the process degrades, any inputs would be appreciated.
The producer parameters are
batch.size=1000000
linger.ms=30000
acks=-1
metadata.fetch.timeout.ms=1000
compression.type=none
max.request.size=10000000
Athough the buffer is fully available the errors are "org.apache.kafka.common.errors.TimeoutException: Failed to allocate memory within the configured max blocking time"
at one point you are starting to send batches of 1.000.000 messages I think that that's why you performance gets degraded. Try lowering that number or set the linger.ms lower.
Test Summary -
Jmeter Version - 2.13
Jmeter Machines - 10 AWS EC2 m4.4xlarge instances
Number of threads on each instance 72 hence in total 720 threads in distributed mode
Test is executed in non GUI mode
I was experimenting with Backend listener as described here and came across drastic reduction in throughput against a static html file. These are the results are received for 5 minutes test -
Throughput with backend listener - 5000/sec
Throughput without backend listener - 9800/sec
I have repeated the test over a period of one week and test results have been consistent.
I did not see any significant difference in load avg or cpu utilization on load agents with or without backend listener.
Is JMeter performance degradation a known issue with Backend listener?
Hmm, interesting. Could it be that the additional time required to do the backend write means that 1 iteration takes a bit longer to complete, which in turn would mean a drop in throughput per thread? Since you're keeping the number of threads constant, your overall throughput would, as a result, drop a bit.
Here's an experiment I would conduct: Turn off backend listener, but put a 500ms constant timer in your test threads. Does it result in an overall throughput drop?
Did you try playing with Async Queue Size 5000 might be very short for a test that has no timer?
The difference can be explained by many factors:
Network quality between JMeter and the InfluxDB or Graphite server
Number of Samples that you track
The queue size, if you test is at high throughput (no timer), then the Async Queue Size will slow down compared to no using backend listener.
Bear in mind also that Increasing Async Queue Size will increase memory footprint
I have a simple configuration for testing: a fast C++ producer sending ~60 byte messages via Stomp to a topic, a slow consumer, and address-full-policy set to DROP.
The queue grows rapidly receiving several thousand messages per second until it reaches my max-size-bytes which amounts to about 300,000 messages. HornetQ starts dropping messages as expected, but from then on is accepting only 3-4 messages per second from the producer. What would cause that? If it's dropping messages, shouldn't it be able to accept them full speed from the producer?
I'm putting a large number of small items to S3 using the REST API. The average payload is ~10 bytes.
The items are all going into one bucket, and have randomized names (i.e. there is no lexicographical order)
From EC2, I've managed a rate of 4-500 per second. I'm using a thread pool of 96 threads, with 64 TCP connections.
I occasionally get an HTTP 500, but have not yet received 503 - meant to indicate that the client slows the rate of requests.
Does anyone know what I can realistically attain? I know the pipe between EC2 and S3 can manage a throughput of 20 MB/s, so I'm hoping to do a bit better.
There should be no surprise that you are seeing poor performance using REST for transfer for such tiny payloads.
The way to do better is to restructure the nature of your protocol or storage so that the transaction overhead doesn't isn't the dominant factor.
Indeed, the size of the pipe is sort of immaterial to your question as you're filling it completely with HTTP overhead; for example if you could double the throughput of the connection, you'd be able to send twice as much useless overhead with effectively no change in usable data.