Kafka streams app not streaming/consuming data after idle time - apache-kafka-streams

Question

I have a Kafka streams app with Processor API
enter co #Bean
#Primary
public KafkaStreams kafkaStreams() {
log.info("Create Kafka Stream Bean with defined topology");
Topology topology = this.buildTopology(new StreamsBuilder());
final KafkaStreams kafkaStreams = new KafkaStreams(topology, createConfigurationProperties());
kafkaStreams.cleanUp();
kafkaStreams.start();
return kafkaStreams;
}
private Topology buildTopology(StreamsBuilder streamsBuilder) {
Topology topology = streamsBuilder.build();
StoreBuilder<KeyValueStore<String, ParticipantObj>> stateStoreBuilder =
Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore("balance"), keySerializer, valuePSerializer);
StoreBuilder<KeyValueStore<String, Long>> lastSSNstateStoreBuilder =
Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore("lastSSN"), keySerializer, valueLongSerializer);
topology.addSource("Source", keyDeSerializer, valueDeSerializer, TopicA, TopicB, TopicC)
.addProcessor("Process", this::getKafkaStreamsNewProcessor, "Source")
.addStateStore(stateStoreBuilder, "Process")
.addStateStore(lastSSNstateStoreBuilder, "Process");
return topology;
}
I have a 3 source topics that I 'm streaming from. I have a processor, 2 state stores(in-memory) and NO sink node. However I do output some data on an outbound topic in the processor itself.
I was able to bring the app up and the data gets consumed for a while. After a while, when there is idle time (after 10 min of idle time) more data is coming from Topics A, B and C but the streams app becomes unresponsive and doesn't consume anything. There are no ERRORS in the logs.
I have enabled DEBUG logs. Still don't see anything in the logs. The app needs to consume data on a daily basis. But data actively comes through the Topics only in certain times of the day. Is there anything else I 'm missing.
Below is my streams config from logs.
acceptable.recovery.lag = 10000
application.id = streams-app1
application.server =
bootstrap.servers = XXXX
buffered.records.per.partition = 1000
built.in.metrics.version = latest
cache.max.bytes.buffering = 10485760
client.id =
commit.interval.ms = 30000
connections.max.idle.ms = 540000
default.deserialization.exception.handler = class org.apache.kafka.streams.errors.LogAndFailExceptionHandler
default.key.serde = class org.apache.kafka.common.serialization.Serdes$StringSerde
default.production.exception.handler = class org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
default.timestamp.extractor = class org.theclearinghouse.chips.kafka.config.MessageTimestampExtractor
default.value.serde = class org.apache.kafka.common.serialization.Serdes$StringSerde
max.task.idle.ms = 0
max.warmup.replicas = 2
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
num.standby.replicas = 0
num.stream.threads = 1
partition.grouper = class org.apache.kafka.streams.processor.DefaultPartitionGrouper
poll.ms = 100
probing.rebalance.interval.ms = 600000
processing.guarantee = at_least_once
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
replication.factor = 1
request.timeout.ms = 40000
retries = 0
retry.backoff.ms = 100
rocksdb.config.setter = null
UPDATE: I do see the below message in the logs..
streams-app1-b3fecd76-3cf1-44ba-90fb-2e07389885c6-StreamThread-1-consumer-08e5aa6d-dedf-4935-a491-6ed96365c0ed sending LeaveGroup request to coordinator bXXXX(id: 2147483644 rack: null) due to consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
2022-08-05 19:40:58,508 [kafka-coordinator-heartbeat-thread | streams-app1] DEBUG org.apache.kafka.clients.NetworkClient - [Consumer clientId=streams-app1-b3fecd76-3cf1-44ba-90fb-2e07389885c6-StreamThread-1-consumer, groupId=streams-app1] Sending LEAVE_GROUP request with header RequestHeader(apiKey=LEAVE_GROUP, apiVersion=4, clientId=streams-ap