Storm bolt connecting to a database - apache-storm

Question

I have a spout which reads from a source with 40K qps.
I have two bolt, first one which reads from the source and does a database connection to build a cache which refreshes in every hour. The database has 2 connection open for a user so executor count that I have for this bolt is 2.
Other bolt is assigned 200 executors and 200 task to process the request.
I can't increase the connection to db. And I see that all the request is going to single workers. Other workers keep waiting and prints "0 send message".
kafkaSpoutConfigList:
- executorsCount: 30
taskCount: 30
spoutName: 'kafka_consumer_spout'
topicName: 'request'
processingBoltConfigList:
- executorsCount: 2
taskCount: 2
boltName: 'db_bolt'
boltClassName: 'com.Bolt1Class'
boltSourceList:
- 'kafka_consumer_spout'
- executorsCount: 200
taskCount: 200
boltName: 'bolt2'
boltClassName: 'com.Bolt2Class'
boltSourceList:
- 'db_bolt::streamx'
kafkaBoltConfigList:
- executorsCount: 15
taskCount: 15
boltName: 'kafka_producer_bolt'
topicName: 'consumer_topic'
boltSourceList:
- 'bolt2::Stream1'
- executorsCount: 15
taskCount: 15
boltName: 'kafka_producer_bolt'
topicName: 'data_test'
boltSourceList:
- 'bolt2::Stream2'
I am using localandgroupshuffling.

Answer 1

When you use LocalOrShuffleGrouping, the following happens:
If the target bolt has one or more tasks in the same worker process, tuples will be shuffled to just those in-process tasks. Otherwise, this acts like a normal shuffle grouping
So let's say your workers look like this:
worker1: {"bolt1 task 1", "bolt2 task 0-50"}
worker2: { "bolt1 task 2", "bolt2 task 50-100"}
worker3: { "bolt2 task 100-150"}
worker4: { "bolt2 task 150-200"}
In this case because you're telling Storm to use a local grouping when sending from bolt1 to bolt2, all the tuples will be going to worker 1 and 2. Worker 3 and 4 will be idle.
If you want to send tuples also to worker 3 and 4, you need to switch to shuffle grouping.