I have created a single queue with daily rolling - chronicle

I have created a single queue with daily rolling. On the next day, I can't read the latest appended message. I found that the tailer index doesn't move to the latest cycle automatically after reading all messages in the previous cycle. By the way the java process was shut down at night and restarted on the next day.
I use Chronicle Queue V4.52.
Thanks.

This should work, we have tests which show messages are read from one cycle to the next.
Would you be able to include a test which reproduces this. There are quite a few unit tests you can use as examples.

this should now be fixed in the latest version
<dependency>
<groupId>net.openhft</groupId>
<artifactId>chronicle-bom</artifactId>
<version>1.13.15</version>
<type>pom</type>
<scope>import</scope>
</dependency>
or if you prefer
<dependency>
<groupId>net.openhft</groupId>
<artifactId>chronicle-queue</artifactId>
<version>4.5.7</version>
</dependency>
also see test case net.openhft.chronicle.queue.impl.single.SingleChronicleQueueTest#testReadingWritingWhenCycleIsSkipped
#Test
public void testReadingWritingWhenCycleIsSkipped() throws Exception {
final Path dir = Files.createTempDirectory("demo");
final RollCycles rollCycle = RollCycles.TEST_SECONDLY;
// write first message
try (ChronicleQueue queue = ChronicleQueueBuilder
.single(dir.toString())
.rollCycle(rollCycle).build()) {
queue.acquireAppender().writeText("first message");
}
Thread.sleep(2100);
// write second message
try (ChronicleQueue queue = ChronicleQueueBuilder
.single(dir.toString())
.rollCycle(rollCycle).build()) {
queue.acquireAppender().writeText("second message");
}
// read both messages
try (ChronicleQueue queue = ChronicleQueueBuilder
.single(dir.toString())
.rollCycle(rollCycle).build()) {
ExcerptTailer tailer = queue.createTailer();
Assert.assertEquals("first message", tailer.readText());
Assert.assertEquals("second message", tailer.readText());
}
}

Related

Stop consumption of message if it cannot be completed

I'm new to mass transit and have a question regarding how I should solve a failure to consume a message. Given the below code I am consuming INotificationRequestContract's. As you can see the code will break and not complete.
public class NotificationConsumerWorker : IConsumer<INotificationRequestContract>
{
private readonly ILogger<NotificationConsumerWorker> _logger;
private readonly INotificationCreator _notificationCreator;
public NotificationConsumerWorker(ILogger<NotificationConsumerWorker> logger, INotificationCreator notificationCreator)
{
_logger = logger;
_notificationCreator = notificationCreator;
}
public Task Consume(ConsumeContext<INotificationRequestContract> context)
{
try
{
throw new Exception("Horrible error");
}
catch (Exception e)
{
// >>>>> insert code here to put message back for later consumption. <<<<<
_logger.LogError(e, "Failed to consume message");
throw;
}
}
}
How do I best handle a scenario such as this where the consumption fails? In my specific case this is likely to occur if a required external service is unavailable.
I can see two solutions.
If there is a way to put the message back, or cancel the consumption so that it will be tried again.
I could store it locally in a database and create my own re-try method to wrap this (but would prefer not to for sake of simplicity).
The exceptions section of the documentation provides sufficient guidance for dealing with consumer exceptions.
There are two retry approaches, which can be used in combination:
Message Retry, which waits while the message is locked, in-process, for the next retry. Therefore, these should be short, to deal with transient issues.
Message Redelivery, which delays the message using either the broker delayed delivery, or a message scheduler, so that it is redelivered to the receive endpoint at some point in the future.
Once all retry/redelivery attempts are exhausted, the message is moved to the _error queue.

Pub Sub Messages still in queue but not pulled

I have a simple shell script that connect to GCP and try to pull Pub/Sub messages from a topic.
When launched, it check if any message exist, does a simple action if so, then ack the message and loop .
It looks like that :
while [ 1 ]
do
gcloud pubsub subscriptions pull...
// Do something
gcloud pubsub subscriptions ack ...
done
Randomly it does not pull the messages : they stay in the queue and are not pulled.
So we tried to add a while loop when getting the message with something like 5 re-try in order to avoid those issues work better but not perfectly. I also think that is a bit shabby...
This issue happened on other project that where migrated from a script shell to Java (for some other reasons) where we used a pull subscription and it work perfectly on those projects now !
We must probably do something wrong but I don't know what...
I have read that sometimes gcloud pull less messages than what's really on the pubsub queue :
https://cloud.google.com/sdk/gcloud/reference/pubsub/subscriptions/pull
But it must at least pull one ... In our case no messages are pulled but randomly.
Is there something to improve here ?
In general, relying on a shell script that uses gcloud to retrieve messages and do something with them is not going to be an efficient way to use Cloud Pub/Sub. It is worth noting that the lack of messages being returned in pull is not indicative of a lack of messages; it just means that messages could not be returned before the pull request's deadline. The gcloud subscriptions pull command sets the returnImmediately property (see info in pull documentation) to true, which basically means that if there aren't messages already quickly accessible in memory, then no messages are going to be returned. This property is deprecated and should not be set to true, so that is probably something that we need to explore changing in gcloud.
You would be better off writing a subscriber using the client libraries that sets up a stream and continuously retrieves messages. If your intention is to run this only periodically, then you could write a job that reads messages and waits some time after messages have not been received and shuts down. Again, this would not guarantee that all messages would be consumed that are available, but it would be true in most cases.
A version of this in Java would look like this:
import com.google.cloud.pubsub.v1.AckReplyConsumer;
import com.google.cloud.pubsub.v1.MessageReceiver;
import com.google.pubsub.v1.ProjectSubscriptionName;
import com.google.pubsub.v1.PubsubMessage;
import java.util.concurrent.atomic.AtomicLong;
import org.joda.time.DateTime;
/** A basic Pub/Sub subscriber for purposes of demonstrating use of the API. */
public class Subscriber implements MessageReceiver {
private final String PROJECT_NAME = "my-project";
private final String SUBSCRIPTION_NAME = "my-subscription";
private com.google.cloud.pubsub.v1.Subscriber subscriber;
private AtomicLong lastReceivedTimestamp = new AtomicLong(0);
private Subscriber() {
ProjectSubscriptionName subscription =
ProjectSubscriptionName.of(PROJECT_NAME, SUBSCRIPTION_NAME);
com.google.cloud.pubsub.v1.Subscriber.Builder builder =
com.google.cloud.pubsub.v1.Subscriber.newBuilder(subscription, this);
try {
this.subscriber = builder.build();
} catch (Exception e) {
System.out.println("Could not create subscriber: " + e);
System.exit(1);
}
}
#Override
public void receiveMessage(PubsubMessage message, AckReplyConsumer consumer) {
// Process message
lastReceivedTimestamp.set(DateTime.now().getMillis());
consumer.ack();
}
private void run() {
subscriber.startAsync();
while (true) {
long now = DateTime.now().getMillis();
long currentReceived = lastReceivedTimestamp.get();
if (currentReceived > 0 && ((now - currentReceived) > 30000)) {
subscriber.stopAsync();
break;
}
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
System.out.println("Error while waiting for completion: " + e);
}
}
System.out.println("Subscriber has not received message in 30s. Stopping.");
subscriber.awaitTerminated();
}
public static void main(String[] args) {
Subscriber s = new Subscriber();
s.run();
System.exit(0);
}
}

Consuming from Camel queue every x minutes

Attempting to implement a way to time my consumer to receive messages from a queue every 30 minutes or so.
For context, I have 20 messages in my error queue until x minutes have passed, then my route consumes all messages on queue and proceeds to 'sleep' until another 30 minutes has passed.
Not sure the best way to implement this, I've tried spring #Scheduled, camel timer, etc and none of it is doing what I'm hoping for. I've been trying to get this to work with route policy but no dice in the correct functionality. It just seems to immediately consume from queue.
Is route policy the correct path or is there something else to use?
The route that reads from the queue will always read any message as quickly as it can.
One thing you could do is start / stop or suspend the route that consumes the messages, so have this sort of set up:
route 1: error_q_reader, which goes from('jms').
route 2: a timed route that fires every 20 mins
route 2 can use a control bus component to start the route.
from('timer?20mins') // or whatever the timer syntax is...
.to("controlbus:route?routeId=route1&action=start")
The tricky part here is knowing when to stop the route. Do you leave it run for 5 mins? Do you want to stop it once the messages are all consumed? There's probably a way to run another route that can check the queue depth (say every 1 min or so), and if it's 0 then shutdown route 1, you might get it to work, but I can assure you this will get messy as you try to deal with a number of async operations.
You could also try something more exotic, like a custom QueueBrowseStrategy which can fire an event to shutdown route 1 when there are no messages on the queue.
I built a customer bean to drain a queue and close, but it's not a very elegant solution, and I'd love to find a better one.
public class TriggeredPollingConsumer {
private ConsumerTemplate consumer;
private Endpoint consumerEndpoint;
private String endpointUri;
private ProducerTemplate producer;
private static final Logger logger = Logger.getLogger( TriggeredPollingConsumer.class );
public TriggeredPollingConsumer() {};
public TriggeredPollingConsumer( ConsumerTemplate consumer, String endpoint, ProducerTemplate producer ) {
this.consumer = consumer;
this.endpointUri = endpoint;
this.producer = producer;
}
public void setConsumer( ConsumerTemplate consumer) {
this.consumer = consumer;
}
public void setProducer( ProducerTemplate producer ) {
this.producer = producer;
}
public void setConsumerEndpoint( Endpoint endpoint ) {
consumerEndpoint = endpoint;
}
public void pollConsumer() throws Exception {
long count = 0;
try {
if ( consumerEndpoint == null ) consumerEndpoint = consumer.getCamelContext().getEndpoint( endpointUri );
logger.debug( "Consuming: " + consumerEndpoint.getEndpointUri() );
consumer.start();
producer.start();
while ( true ) {
logger.trace("Awaiting message: " + ++count );
Exchange exchange = consumer.receive( consumerEndpoint, 60000 );
if ( exchange == null ) break;
logger.trace("Processing message: " + count );
producer.send( exchange );
consumer.doneUoW( exchange );
logger.trace("Processed message: " + count );
}
producer.stop();
consumer.stop();
logger.debug( "Consumed " + (count - 1) + " message" + ( count == 2 ? "." : "s." ) );
} catch ( Throwable t ) {
logger.error("Something went wrong!", t );
throw t;
}
}
}
You configure the bean, and then call the bean method from your timer, and configure a direct route to process the entries from the queue.
from("timer:...")
.beanRef("consumerBean", "pollConsumer");
from("direct:myRoute")
.to(...);
It will then read everything in the queue, and stop as soon as no entries arrive within a minute. You might want to reduce the minute, but I found a second meant that if JMS as a bit slow, it would time out halfway through draining the queue.
I've also been looking at the sjms-batch component, and how it might be used with with a pollEnrich pattern, but so far I haven't been able to get that to work.
I have solved that by using my application as a CronJob in a MicroServices approach, and to give it the power of gracefully shutting itself down, we may set the property camel.springboot.duration-max-idle-seconds. Thus, your JMS consumer route keeps simple.
Another approach would be to declare a route to control the "lifecycle" (start, sleep and resume) of your JMS consumer route.
I would strongly suggest that you use the first approach.
If you use ActiveMQ you can leverage the Scheduler feature of it.
You can delay the delivery of a message on the broker by simply set the JMS property AMQ_SCHEDULED_DELAY to the number of milliseconds of the delay. Very easy in the Camel route
.setHeader("AMQ_SCHEDULED_DELAY", 60000)
It is not exactly what you look for because it does not drain a queue every 30 minutes, but instead delays every individual message for 30 minutes.
Notice that you have to enable the schedulerSupport in your broker configuration. Otherwise the delay properties are ignored.
<broker brokerName="localhost" dataDirectory="${activemq.data}" schedulerSupport="true">
...
</broker>
You should consider Aggregation EIP
from(URI_WAITING_QUEUE)
.aggregate(new GroupedExchangeAggregationStrategy())
.constant(true)
.completionInterval(TIMEOUT)
.to(URI_PROCESSING_BATCH_OF_EXCEPTIONS);
This example describes the following rules: all incoming in URI_WAITING_QUEUE objects will be grouped into List. constant(true) is a grouping condition (wihout any). And every TIMEOUT period (in millis) all grouped objects will be passed into URI_PROCESSING_BATCH_OF_EXCEPTIONS queue.
So the URI_PROCESSING_BATCH_OF_EXCEPTIONS queue will deal with List of objects to process. You can apply Split EIP to split them and to process one by one.

Error while adding nodes and properties using jcr apis Adobe Experience Manager6.0

Using JCR apis while I am trying to add node and property.I am getting the following error:
7520 [main] ERROR org.apache.jackrabbit.jcr2spi.hierarchy.ChildNodeEntriesImpl - ChildInfo iterator contains multiple entries with the same name|index or uniqueID -> ignore ChildNodeInfo.
I have added the following dependency in Pom.xml:
<dependency>
<groupId>org.apache.jackrabbit</groupId>
<artifactId>jackrabbit-jcr-commons</artifactId>
<version>2.12.1</version></dependency>
<dependency>
<groupId>org.apache.jackrabbit</groupId>
<artifactId>jackrabbit-jcr2dav</artifactId>
<version>2.0-beta6</version> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.5.8</version></dependency>
Java code:
package com.adobe.cq.impl;
import javax.jcr.Node;
import javax.jcr.Repository;
import javax.jcr.Session;
import javax.jcr.SimpleCredentials;
import org.apache.jackrabbit.commons.JcrUtils;
public class GetRepository {
public static void main(String[] args) {
try {
Repository repository = JcrUtils.getRepository("http://localhost:4502/crx/server");
Session session=repository.login(new SimpleCredentials("admin", "admin".toCharArray()));
Node root=session.getRootNode();
Node adobe = root.addNode("adobe");
Node day = adobe.addNode("cq");
day.setProperty("message", "Adobe Experience Manager is part of the Adobe Digital Marketing Suite!");
// Retrieve content
Node node = root.getNode("adobe/cq");
System.out.println(node.getPath());
System.out.println(node.getProperty("message").getString());
// Save the session changes and log out
session.save();
session.logout();
}
catch(Exception e){
e.printStackTrace();
}
}}
Same name siblings are not allowed in the repository. Going by your code, there is no check if the node "adobe" is already present below the root node. Hence, if the node is / was already created / present and the above code executes for the second time, you may face this issue.
Try checking for node availability as shown below.
Node adobe;
if (!root.hasNode("adobe")) {
adobe = root.addNode("adobe");
} else {
adobe = root.getNode("adobe");
}
if (!adobe.hasNode("cq")) {
Node day = adobe.addNode("cq");
}

Oracle AQ load test with Gatling

I want to set up a Gatling testcase to put messages on a Oracle AQ. But I have no idea how to set up the the following:
val jmsConfig = jms
.connectionFactoryName(???)
.url("tcp://localhost:10002")
.credentials("user", "secret")
.contextFactory(???)
.listenerCount(1)
.usePersistentDeliveryMode
What is the connection factory name and what is the context factory?
I managed to get it working using the oracle.jms.AQjmsInitialContextFactory. The above mentioned oracle.jms.AQjmsFactory is not an InitialContextFactory so that wont work.
Make sure to add at least version 11+ of the Oracle AQ dependencies for the AQjmsInitialContextFactory to be found.
You database user should of-course have the correct privileges to be able to insert messages into the queue (table).
Gatling expects your to have request-reply semantics, so it will wait for a reply to be received. I actually wanted to abort waiting for a reply after a specified period, but I have no clue how to do that. So if anyone know how to.. please tell me :-)
MySimulation.scala
val jmsConfig = jms
.connectionFactoryName("ConnectionFactory") // MUST!!!! be called ConnectionFactory, the AQjmsFactory expects this naming convention!
.url("jdbc:oracle:thin:#host:1521:SID")
.credentials("user", "password")
.contextFactory("oracle.jms.AQjmsInitialContextFactory")
.listenerCount(1)
.usePersistentDeliveryMode
// TODO check how to set a timeout on the reply
val jmsScenario = scenario("JMS DSL test")
.repeat(1) {
exec(
jms("req reply testing")
.reqreply
.queue("AQ_ADMIN.QUEUE_NAME")
.textMessage("some message")
.check(simpleCheck(checkBodyTextCorrect))
)
}
def checkBodyTextCorrect(m: Message) = {
// this assumes that the service just does an "uppercase" transform on the text
m match {
case tm: TextMessage => tm.getText == "text that should be in the reply message"
case _ => false
}
}
setUp(jmsScenario.inject(atOnceUsers(1)).protocols(jmsConfig));
jndi.properties
I had to add the jndi.properties to the classpath:
db_url=jdbc:oracle:thin:#host:1521:SID
pom.xml
Dependencies (maven):
<dependency>
<groupId>oracle</groupId>
<artifactId>aqapi</artifactId>
<version>11.2.0.3</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>11.2.0.2.0</version>
</dependency>
<dependency>
<groupId>javax.transaction</groupId>
<artifactId>jta</artifactId>
<version>1.1</version>
</dependency>
"contextFactory" is the class name of your ContextFactory. Doc seems to state it's "oracle.jms.AQjmsFactory".
"connectionFactoryName" is the key used for the JNDI look-up. Doc again seems to state it's "cn=OracleDBConnections".

Resources