Tuning my Apache Storm serializer for performance - performance

Question

i am new to Java and Apache Storm and i want to know how i can make things go faster!
I setup a Storm cluster with 2 physical machines with 8 cores each. The cluster is working perfectly fine. I setup the following test topology in order to measure performance:
builder.setSpout("spout", new RandomNumberSpoutSingle(sizeOfArray), 10);
builder.setBolt("null", new NullBolt(), 4).allGrouping("spout");
RandomNumberSpoutSingle creates an Array like so:
ArrayList<Integer> array = new ArrayList<Integer>();
I fill it with sizeOfArray integers. This array, combined with an ID, builds my tuple.
Now i measure how many tuples per second arrive at the bolt with allGrouping (i look at the Storm GUI's "transferred" value).
If i put sizeOfArray = 1024, about 173000 tuples/s get pushed. Since 1 tuple should be about 4*1024 bytes, around 675MB/second get moved.
Am i correct so far?
Now my question is: Is Storm/Kryo capable of moving more? How can i tune this? Are there settings i ignored?
I want to serialize more tuples per second! If i use local shuffling, the values skyrocket because nothing has to be serialized, but i need the tuples on all workers.
Neither CPU, Memory nor network are fully occupied.

Answer 1

I think you got the math about right, I am not sure though if the Java overhead for the non-primitive Integer type is considered in serialization, which would add some more bytes to the equation. Yet, I am also not sure if this is the best way of analyzing storm performance, as this is more measured in number of tuples per second than in bandwidth.
Storm has built in serialization for primitive types, strings, byte arrays, ArrayList, HashMap, and HashSet (source). When I program Java for maximum performance I try to stick with primitive types as much as possible. Would it be feasible to use int[] instead of ArrayList<Integer>? I would expect to gain some performance from that, if it is possible in your setup.
Considering the above types which storm is able to serialize out-of-the-box I would most likely shy away from trying to improve serialization performance. I assume kryo is pretty optimized and that it will be very hard to achieve anything faster here. I am also not sure if serialization is the real bottleneck here or rather something in your topology setup (see below).
I would look at other tunables which are related to the intra and inter worker communication. A good overview can be found here. In one topology for which performance is critical, I am using the following setup code to adjust these kind of parameters. What works best in your case needs to be found out via testing.
int topology_executor_receive_buffer_size = 32768; // intra-worker messaging, default: 32768
int topology_transfer_buffer_size = 2048; // inter-worker messaging, default: 1000
int topology_producer_batch_size = 10; // intra-worker batch, default: 1
int topology_transfer_batch_size = 20; // inter-worker batch, default: 1
int topology_batch_flush_interval_millis = 10; // flush tuple creation ms, default: 1
double topology_stats_sample_rate = 0.001; // calculate metrics every 1000 messages, default: 0.05
conf.put("topology.executor.receive.buffer.size", topology_executor_receive_buffer_size);
conf.put("topology.transfer.buffer.size", topology_transfer_buffer_size);
conf.put("topology.producer.batch.size", topology_producer_batch_size);
conf.put("topology.transfer.batch.size", topology_transfer_batch_size);
conf.put("topology.batch.flush.interval.millis", topology_batch_flush_interval_millis);
conf.put("topology.stats.sample.rate", topology_stats_sample_rate);
As you have noticed, performance greatly increases when storm is able to use intra-worker processing, so I would always suggest to make use of that if possible. Are you sure you need allGrouping? If not I would suggest to use shuffleGrouping, which will actually use local communication if storm thinks it is appropriate, unless topology.disable.loadaware.messaging is set to false. I am not sure if allGrouping will use local communication for those components which are on the same worker.
Another thing which I wonder about is the configuration of your topology: you have 10 spouts and 4 consumer bolts. Unless the bolts consume incoming tuples much faster than they are created, it might be advisable to use an equal number for both components. From how you describe your process it seems you use acking and failing, because you have written you assign an ID to your tuples. In case that guaranteed processing of individual tuples is not a absolute requirement, performance can probably be gained by switching to unanchored tuples. Acking and failing does produce some overhead, so I would assume a higher tuple throughput if it is turned off.
And lastly, you can also experiment with the value for maximum number of pending tuples (configured via method .setMaxSpoutPending of the spouts). Not sure what storm uses as default, however from my experience setting a little higher number than what the bolts can ingest downstream delivers higher throughput. Look at metrics capacity and number of transferred tuples in the storm UI.