ActiveMQ not freeing up the disk storage for queue - spring

Question

When throwing a significant amount of load to an ActiveMQ queue it seems to enqueue/dequeue messages as it's supposed to, but the storage remains full.
This causes the producer to block after some time, when the storage gets to 100%, which in turn causes the application to stop responding to requests.
This is the relevant activeMQ Spring configuration that we're using:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- =====================================================
Broker Configuration
===================================================== -->
<broker id="appEmbeddedBroker"
xmlns="http://activemq.apache.org/schema/core"
brokerName="${msg.embedded.broker.name}"
persistent="true"
dataDirectory="${msg.embedded.broker.data.directory}"
useJmx="true" >
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext connectorPort="${msg.embedded.broker.jmx.port}" createConnector="false"/>
</managementContext>
<persistenceAdapter>
<levelDB directory="${msg.embedded.broker.db.directory}" />
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="10"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="${msg.embedded.broker.system.usage.store.usage}"/> <!-- Configured for 200Mb -->
</storeUsage>
<tempUsage>
<tempUsage limit="${msg.embedded.broker.system.usage.temp.usage}"/> <!-- Configured for 40Mb -->
</tempUsage>
</systemUsage>
</systemUsage>
<plugins>
<!-- Configure authentication; Username, passwords and groups -->
<simpleAuthenticationPlugin anonymousAccessAllowed="false">
<users>
<authenticationUser username="app" password="${msg.embedded.broker.app.password}"
groups="users"/>
</users>
</simpleAuthenticationPlugin>
</plugins>
<transportConnectors>
<transportConnector name="tcp" uri="tcp://0.0.0.0:${msg.embedded.broker.port}?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
<!-- =====================================================
Client Connections
===================================================== -->
<bean id="embeddedAmqConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory" depends-on="appEmbeddedBroker" >
<property name="brokerURL" value="${msg.embedded.broker.url}" />
<property name="userName" value="${msg.embedded.client.app.username}" />
<property name="password" value="${msg.embedded.broker.app.password}" />
</bean>
<bean id="embeddedAmqPooledConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" >
<property name="connectionFactory" ref="embeddedAmqConnectionFactory" />
<property name="maxConnections" value="${msg.embedded.client.connection.pool.max}" />
</bean>
When querying the beans:
$>get QueueSize EnqueueCount DequeueCount
#mbean = org.apache.activemq:brokerName=app-embedded,destinationName=the.queue.name.local,destinationType=Queue,type=Broker:
QueueSize = 0;
EnqueueCount = 17821;
DequeueCount = 17821;
And:
#mbean = org.apache.activemq:brokerName=app-embedded,type=Broker:
StoreLimit = 209715200;
StorePercentUsage = 100;
TempLimit = 41943040;
TempPercentUsage = 0;
So, the question would be: Why if all messages are getting dequeued we still see 100% of storage usage?

Answer 1

There can be many reasons for this depending on you broker configuration and usage. First thing to check is what the contents of your DLQ are, then look on the console for other destinations that are holding onto messages. Due to the way things are propagated through the KahaDB journal logs it is possible that they create a chain that cannot be broken until you purge some other Queues or rollback any Transactions that are still inflight etc.
There is a guide on the ActiveMQ site that covers how to start debugging the situation that should help get you started.
Also try running the latest broker release, there are fixes that have gone into these components that try and address these sorts of situations.