I'm having an issue creating a RabbitMQ endpoint with Camel. The issue only occurs when I declare a a dead message letter exchange option based on the camel documentation. This is my URN for creating the endpoint:
rabbitmq://localhost/com.mydomain.inbound.exhange?deadLetterExchange=dead.msgs
All is fine when I omit the deadLetterExchange option but as soon as I include it I get the following (not very helpful) exception:
Caused by: java.lang.NullPointerException
at com.rabbitmq.client.impl.ChannelN.validateQueueNameLength(ChannelN.java:1244) ~[amqp-client-3.6.1.jar:?]
at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:843) ~[amqp-client-3.6.1.jar:?]
at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:61) ~[amqp-client-3.6.1.jar:?]
at org.apache.camel.component.rabbitmq.RabbitMQDeclareSupport.declareAndBindQueue(RabbitMQDeclareSupport.java:96) ~[camel-rabbitmq-2.17.0.jar:2.17.0]
at org.apache.camel.component.rabbitmq.RabbitMQDeclareSupport.declareAndBindDeadLetterExchangeWithQueue(RabbitMQDeclareSupport.java:43) ~[camel-rabbitmq-2.17.0.jar:2.17.0]
at org.apache.camel.component.rabbitmq.RabbitMQDeclareSupport.declareAndBindExchangesAndQueuesUsing(RabbitMQDeclareSupport.java:35) ~[camel-rabbitmq-2.17.0.jar:2.17.0]
at org.apache.camel.component.rabbitmq.RabbitMQEndpoint.declareExchangeAndQueue(RabbitMQEndpoint.java:222) ~[camel-rabbitmq-2.17.0.jar:2.17.0]
at org.apache.camel.component.rabbitmq.RabbitConsumer.openChannel(RabbitConsumer.java:288) ~[camel-rabbitmq-2.17.0.jar:2.17.0]
at org.apache.camel.component.rabbitmq.RabbitConsumer.(RabbitConsumer.java:57) ~[camel-rabbitmq-2.17.0.jar:2.17.0]
at org.apache.camel.component.rabbitmq.RabbitMQConsumer.createConsumer(RabbitMQConsumer.java:108) ~[camel-rabbitmq-2.17.0.jar:2.17.0]
at org.apache.camel.component.rabbitmq.RabbitMQConsumer.startConsumers(RabbitMQConsumer.java:90) ~[camel-rabbitmq-2.17.0.jar:2.17.0]
at org.apache.camel.component.rabbitmq.RabbitMQConsumer.doStart(RabbitMQConsumer.java:160) ~[camel-rabbitmq-2.17.0.jar:2.17.0]
at org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:61) ~[camel-core-2.17.0.jar:2.17.0]
at org.apache.camel.impl.DefaultCamelContext.startService(DefaultCamelContext.java:3269) ~[camel-core-2.17.0.jar:2.17.0]
at org.apache.camel.impl.DefaultCamelContext.doStartOrResumeRouteConsumers(DefaultCamelContext.java:3563) ~[camel-core-2.17.0.jar:2.17.0]
at org.apache.camel.impl.DefaultCamelContext.doStartRouteConsumers
....
Just no note that I've also tried creating the exchange and queue manually in a hope that this may work but no luck.
Additional Info:
camel-spring-boot-starter (2.17.0)
camel-rabbitmq (2.17.0)
Try adding a deadLetterQueueoption -
rabbitmq://localhost/com.mydomain.inbound.exhange?deadLetterExchange=dead.msgs&deadLetterQueue=my.dead.letter.queue
I also had to add further options to the uri to get it to work. I added
deadLetterExchangeType
queueArgsConfigurer
The queueArgsConfigurer is an implementation of org.apache.camel.component.rabbitmq.ArgsConfigurer
class MyQueueArgs implements ArgsConfigurer {
void configurArgs(Map<String, Object> args) { //misspelling!!
args.put("x-dead-letter-exchange", "my.dead.letter")
args.put("x-dead-letter-routing-key", "my.dead.letter.key")
}
}
Mine is a Spring app so myArgs (see below) is created in the bean factory.
So, the full uri is like this -
rabbitmq://hostname/exchangeName?routingKey=$routingKey&vhost=virtualHostname&exchangeType=exType&autoDelete=false&queue=my.queue&deadLetterExchange=my.dead.letter&deadLetterExchangeType=dlExType&deadLetterQueue=my.dead.letter.queue&queueArgsConfigurer=#myArgs
I probably don't need to specify the dead letter exchange in the uri and the ArgsConfigurer implementation.
For more on ArgsConfigurer this Camel issue might help - #8457
I had to look at the source code to figure a lot of this out. What is missing from the doc is a definition of dependencies. There are some options, particularly around dead letter exchanges, which become mandatory if another is specified. That's why you are getting your errors. Have a look at populateQueueArgumentsFromDeadLetterExchange in RabbitMQDeclareSupport.
EDIT
A simplification to my answer - I dropped the ArgsConfigurer implementation in the end. I went with this -
rabbitmq://myHostname/myExchangeName?
username=myUserName&
password=myPassword&
queue=myQueueName&
routingKey=myRoutingKey&
vhost=myVirtualHostname&
exchangeType=topic&
autoDelete=false&
deadLetterExchange=myDeadLetter&
deadLetterExchangeType=topic&
deadLetterQueue=myDeadLetterQueue&
deadLetterRoutingKey=myDeadLetterRoutingKey&
autoAck=false
Related
That's not really an issue, beacause I found a workaround, but it conflicts with the documentation, so I wanted to share and document about it.
FYI Spring Boot 2.1.10 + SCSt 2.1.4 + RabbitMQ binder
I first implemented an application local error handler as given into official docs :
#StreamListener(Sink.INPUT)
public void handle(Person value) {
throw new RuntimeException("BOOM!");
}
#ServiceActivator(inputChannel = Sink.INPUT + ".my-group.errors") // won't work
public void error(ErrorMessage message) {
log.error("Handling ERROR: " + message.getPayload().getMessage());
}
spring.cloud.stream.bindings.input.destination=persons.inputs
spring.cloud.stream.bindings.input.group=my-group
But that didn't go well, to say the least. This is what I eventually had to keep:
#ServiceActivator(inputChannel = "persons.inputs.my-group.errors")
As you can see, what's happening is that I had to stick to the actual destination definition instead of the channel's; which I think is very uncomfortable! And I want to underline, again, that this is contradictory to the official docs here: https://docs.spring.io/spring-cloud-stream/docs/current/reference/htmlsingle/#_application_error_handling (plus there are noticeable typos, IMHO: they even write that the destinationName is actually required)
Can anyone share thoughts about the situation with me? Have I done it right and am I right to think that this is wrong?
It's a bug in the documentation; it is, indeed, unfortunate the binding name was not used in the error channel name instead of the destination and group, but it's too late to change it now. We could possibly do something in a future release.
Please open 2 GitHub issues to
fix the documentation
consider adding an option to name the error channel using the binding name instead.
Overview.
My camel setup calls two service methods. the response of the first one is passed into the second one and then output the final response as json webpage. Fairly simple nothing too complicated.
Further breakdown to give some more context.
Method_1. Takes in scanId. This works ok. It produces an object called ScheduledScan .class
Method_2. Takes in object previous instance of ScheduledScan .class and returns a list of ConvertedScans scans. Then would like to display said list
Description of the code
#Override
public void configure() throws Exception {
restConfiguration().bindingMode(RestBindingMode.json);
rest("/publish")
.get("/scheduled-scan/{scanId}")
.to("bean:SentinelImportService?method=getScheduledScan").outType(ScheduledScan .class)
.to("bean:SentinelImportService?method=convertScheduledScan");
}
The methods that are called look like the following
ScheduledScan getScheduledScan(#Header("scanId") long scanId);
List<ConvertedScans > convertScheduledScan(#Body ScheduledScan scheduledScans);
It is returning the the following error
No body available of type: path. .ScheduledScan but has value:
of type: java.lang.String on: HttpMessage#0x63c2fd04. Caused by: No type converter available
The following runs without the error, i.e. without method 2. So I think im almost there.
rest("/publish")
.get("/scheduled-scan/{scanId}")
.to("bean:SentinelImportService?method=getScheduledScan");
Now from reading the error it looks like im passing in a HttpMessage not the java object? I'm a bit confused about what to do next? Any advice much appreciated.
I have found some similar questions to this message. However I am looking to pass the java object directly into the service method.
camel-rest-bean-chaining
how-to-share-an-object-between-methods-on-different-camel-routes
You should setup the outType as the last output, eg what the REST response is, that is a List/Array and not a single pojo. So use .outTypeList(ConvertedScans.class) instead.
I just started using the driver org.apache.hive.jdbc.HiveDriver (Version
1.2.1 for spark2) with a Spark Thrift Server (STS) (reference here)
java.sql.ResultSet defines the method absolute() (JavaDoc here)
but HiveBaseResultSet seems to have chosen not to implement the method (source code here)
So now my application (built on top of SmartGWT) was doing a simple operation and I got the following error message:
=== 2017-05-13 18:06:16,980 [3-47] WARN RequestContext - dsRequest.execute() failed:
java.sql.SQLException: Method not supported
at org.apache.hive.jdbc.HiveBaseResultSet.absolute(HiveBaseResultSet.java:70)
at org.apache.commons.dbcp.DelegatingResultSet.absolute(DelegatingResultSet.java:373)
at com.isomorphic.sql.SQLDataSource.executeWindowedSelect(SQLDataSource.java:2970)
at com.isomorphic.sql.SQLDataSource.SQLExecute(SQLDataSource.java:2024)
What is the reason that the driver chose not to implement absolute()?
Are there any workaround for the limitation?
Thanks for the hint from Mark Rotteveel. Now I understand better and let me post an answer to my own question.
Implementation of absolute() is optional
As specified by the Interface of ResultSet#absolute() (link), the implementation for absolute() is optional -- especially when the result set type is TYPE_FORWARD_ONLY.
Workaround
In my case, the result set comes from a Spark Thrift Server (STS) so I guess it is indeed forward-only. So the question became how to instruct my application to NOT making a call to absolute(), which is basically for cursor movement.
SmartGWT-specific answer
For SmartGWT, this is controlled by a property called sqlPaging, which we can specified for an OperationBinding. The right value to use seems to be dropAtServer (more reference here). So I set my SmartGWT DataSource XML file to something like this
<operationBindings>
<operationBinding operationType="fetch" progressiveLoading="false"
sqlPaging="dropAtServer"
>
After that I saw another error, which is now related to HiveConnection#commit():
java.sql.SQLException: Method not supported
at org.apache.hive.jdbc.HiveConnection.commit(HiveConnection.java:742)
at org.apache.commons.dbcp.DelegatingConnection.commit(DelegatingConnection.java:334)
at com.isomorphic.sql.SQLTransaction.commitTransaction(SQLTransaction.java:307)
at com.isomorphic.sql.SQLDataSource.commit(SQLDataSource.java:4673)
After more digging, I realized that the right property for SmartGWT to control the commit behavior is autoJoinTransactions and I should set it to false (more reference here). After these two changes, I could get may application to talk to STS via jdbc.HiveDriver
For anyone out there who are also trying this, here is my full settings for the driver in SmartGWT's server.properties (more reference here)
sql.defaultDatabase: perf2 # this name is picked by me, but it can be anyname
sql.perf2.driver.networkProtocol: tcp
sql.perf2.driver: org.apache.hive.jdbc.HiveDriver # important
sql.perf2.database.type: generic # important
sql.perf2.autoJoinTransactions: false # important
sql.perf2.interface.type: driverManager # important
sql.perf2.driver.url: jdbc:hive2://host:port # important -- pick your host:port
sql.perf2.driver.user: someuser # important -- pick your username
sql.perf2.interface.credentialsInURL: true
sql.perf2.driver.databaseName: someDb
sql.perf2.driver.context:
Is it possible to extend the visibility time out of a message that is in flight.
See:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/AboutVT.html.
Section: Changing a Message's Visibility Timeout.
http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/sqs/AmazonSQSClient.html#changeMessageVisibility-com.amazonaws.services.sqs.model.ChangeMessageVisibilityRequest-
In summary I want to be able to extend the first set visibility timeout for a given message that is in flight.
Example if 15secs have passed I then want to extend the timeout by another 20secs. Better example in java docs above.
From my understanding in the links above you can do this on the amazon side.
Below are my current settings;
SqsMessageDrivenChannelAdapter adapter =
new SqsMessageDrivenChannelAdapter(queue);
adapter.setMessageDeletionPolicy(SqsMessageDeletionPolicy.ON_SUCCESS);
adapter.setMaxNumberOfMessages(1);
adapter.setSendTimeout(2000);
adapter.setVisibilityTimeout(200);
adapter.setWaitTimeOut(20);
Is it possible to extend this timeout?
Spring Cloud AWS supports this starting with Version 2.0. Injecting a Visiblity parameter in your SQS listener method does the trick:
#SqsListener(value = "my-sqs-queue")
void onMessageReceived(#Payload String payload, Visibility visibility) {
...
var extension = visibility.extend(20);
...
}
Note, that extend will work asynchronously and will return a Future. So if you want to be sure further down the processing, that the visibility of the message is really extended at the AWS side of things, either block on the Future using extension.get() or query the Future with extension.isDone()
OK. Looks like I see your point.
We can change visibility for particular message using API:
AmazonSQS.changeMessageVisibility(String queueUrl, String receiptHandle, Integer visibilityTimeout)
For this purpose in downstream flow you have to get access to (inject) AmazonSQS bean and extract special headers from the Message:
#Autowired
AmazonSQS amazonSqs;
#Autowired
ResourceIdResolver resourceIdResolver;
...
MessageHeaders headers = message.getHeaders();
DestinationResolver destinationResolver = new DynamicQueueUrlDestinationResolver(this.amazonSqs, this.resourceIdResolver);
String queueUrl = destinationResolver.resolveDestination(headers.get(AwsHeaders.QUEUE));
String receiptHandle = headers.get(AwsHeaders.RECEIPT_HANDLE);
amazonSqs.changeMessageVisibility(queueUrl, receiptHandle, YOUR_DESIRED_VISIBILITY_TIMEOUT);
But eh, I agree that we should provide something on the matter as out-of-the-box feature. That may be even something similar to QueueMessageAcknowledgment as a new header. Or even just one more changeMessageVisibility method to this one.
Please, raise a GH issue for Spring Cloud AWS project on the matter with link to this SO topic.
I'm seeing a very strange behavior in my application.
My application setup: Spring + Hibernate + C3p0
Application keeps running fine, when all of a sudden I start seeing these errors in logs and system totally stop processing any database specific requests.
WARN c3p0.C3P0Registry - Could not create for find ConnectionCustomizer with class name ''.
java.lang.ClassNotFoundException:
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at com.mchange.v2.c3p0.C3P0Registry.getConnectionCustomizer(C3P0Registry.java:181)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolManager.getConnectionCustomizer(C3P0PooledConnectionPoolManager.java:636)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolManager.createPooledConnectionPool(C3P0PooledConnectionPoolManager.java:738)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolManager.getPool(C3P0PooledConnectionPoolManager.java:257)
at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPoolManager.getPool(C3P0PooledConnectionPoolManager.java:271)
at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:80)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.AbstractBatcher.prepareSelectStatement(AbstractBatcher.java:123)
at org.hibernate.id.SequenceGenerator.generate(SequenceGenerator.java:73)
at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:99)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.saveWithGeneratedOrRequestedId(DefaultSaveOrUpdateEventListener.java:187)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.entityIsTransient(DefaultSaveOrUpdateEventListener.java:172)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.performSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:94)
at org.hibernate.event.def.DefaultSaveOrUpdateEventListener.onSaveOrUpdate(DefaultSaveOrUpdateEventListener.java:70)
at org.hibernate.impl.SessionImpl.fireSaveOrUpdate(SessionImpl.java:507)
at org.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:499)
at org.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:495)
at org.springframework.orm.hibernate3.HibernateTemplate$18.doInHibernate(HibernateTemplate.java:690)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:365)
at org.springframework.orm.hibernate3.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:687)
Why would C3p0 require to create a new connection pool at this
particular time, before these exceptions application is 100% working
fine and responding perfectly.
Also I've not provided any connectionCustomizerClassName property in
my c3p0 configurations, why would it load one? in this stack trace I
see it's not-null empty string ''.
Any clues?
==============================================================================
Following hibernate jars I see in application's classpath:
hibernate-3.2.6.ga.jar
spring-hibernate-1.2.6.jar
Following c3p0 jars I see in application's classpath:
c3p0-0.9.1.jar
c3p0-0.9.2-pre5.jar
c3p0-oracle-thin-extras-0.9.2-pre5.jar
Code that manually read these properties and set on datasource (I do not read/set any connectionCustomizerClassName property here at all)
ComboPooledDataSource dataSource = new ComboPooledDataSource();
dataSource.setMinPoolSize(Integer.parseInt(props.getProperty("jdbc.hibernate.c3p0.minPoolSize")));
.....
Here are C3p0 properties being used:
jdbc.hibernate.c3p0.minPoolSize=100
jdbc.hibernate.c3p0.initialPoolSize=100
jdbc.hibernate.c3p0.maxPoolSize=1000
jdbc.hibernate.c3p0.maxIdleTime=21600
jdbc.hibernate.c3p0.maxStatementsPerConnection=0
jdbc.hibernate.c3p0.maxStatements=0
jdbc.hibernate.c3p0.numHelperThreads=30
jdbc.hibernate.c3p0.checkoutTimeout=30000
jdbc.hibernate.c3p0.idleConnectionTestPeriod=900
jdbc.hibernate.c3p0.preferredTestQuery=SELECT 1 FROM dual
jdbc.hibernate.c3p0.maxConnectionAge=0
jdbc.hibernate.c3p0.maxIdleTimeExcessConnections=3600
jdbc.hibernate.c3p0.acquireIncrement=10
jdbc.hibernate.c3p0.acquireRetryDelay=5000
jdbc.hibernate.c3p0.acquireRetryAttempts=6
jdbc.hibernate.c3p0.propertyCycle=180
Following up a conversation in the comments on the posted question, it looks like the issue here is that VisualVM updates the null valued property connectionCustomizerClassName to an empty String value, which c3p0 currently treats an non-null and interprets as a class name.
Going forward (c3p0-0.9.5-pre7 and above), c3p0 will guard against this, interpret an all-whitespace connectionCustomizerClassName as equivalent to null. But in the meantime or for older versions, take care.
One easy workaround would be to define a NullConnectionCustomizer:
package mypkg;
import com.mchange.v2.c3p0.*;
public class NullConnectionCustomizer extends AbstractConnectionCustomizer
{}
And then use mypkg.NullConnectionCustomizer for connectionCustomizerClassName, so that the corresponding field in VisualVM is not empty and ambiguously interpretable as empty String or null.