Apache Camel has a number of features which make event processing elegant and easy to code. It would be useful to be able to exploit this in an AWS Lambda.
Of course not all features are appropriate, especially anything requiring a long lived process.
Also managing persistant state, for example idempotent repositories and throttling would need thinkng about.
But it would be really useful in simple cases.
It turns out that this is simple using Redhat's Quarkus framework.
I've made a simple example: https://github.com/jcable/SampleCamelLambda
The Camel Route is trivial:
from("direct:input").to("log:input")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
InputObject input = exchange.getIn().getBody(InputObject.class);
String result = input.getGreeting() + " " + input.getName();
OutputObject out = new OutputObject();
out.setResult(result);
out.setRequestId("aws-request-1");
exchange.getIn().setBody(out);
}
});
Adapting the route to the Lambda makes use of a Quarkus RequestHandler.
public class Lambda implements RequestHandler<InputObject, OutputObject> {
#Inject
CamelContext camelContext;
#Override
public OutputObject handleRequest(InputObject input, Context context) {
return camelContext.createProducerTemplate().requestBody("direct:input", input, OutputObject.class);
}
}
CDI is used to inject the CamelContext into the request handler and then the camelContext object is used to create a
ProducerTemplate which can be used to invoke the Camel route.
The Maven project for the example is derived from the Quarkus lambda example with Apache Camel dependencies from the Camel Quarkus examples.
Related
I was reading "Spring Microservices In Action (2021)" because I wanted to brush up on Microservices.
Now with Spring Boot 3 a few things changed. In the book, an easy example of how to push messages to a topic and how to consume messages to a topic were presented.
The Problem is: The examples presented do just not work with Spring Boot 3. Sending Messages from a Spring Boot 2 Project works. The underlying project can be found here:
https://github.com/ihuaylupo/manning-smia/tree/master/chapter10
Example 1 (organization-service):
Consider this Config:
spring.cloud.stream.bindings.output.destination=orgChangeTopic
spring.cloud.stream.bindings.output.content-type=application/json
spring.cloud.stream.kafka.binder.zkNodes=kafka #kafka is used as a network alias in docker-compose
spring.cloud.stream.kafka.binder.brokers=kafka
And this Component(Class) which can is injected in a service in this project
#Component
public class SimpleSourceBean {
private Source source;
private static final Logger logger = LoggerFactory.getLogger(SimpleSourceBean.class);
#Autowired
public SimpleSourceBean(Source source){
this.source = source;
}
public void publishOrganizationChange(String action, String organizationId){
logger.debug("Sending Kafka message {} for Organization Id: {}", action, organizationId);
OrganizationChangeModel change = new OrganizationChangeModel(
OrganizationChangeModel.class.getTypeName(),
action,
organizationId,
UserContext.getCorrelationId());
source.output().send(MessageBuilder.withPayload(change).build());
}
}
This code fires a message to the topic (destination) orgChangeTopic. The way I understand it, the firsttime a message is fired, the topic is created.
Question 1: How do I do this Spring Boot 3? Config-Wise and "Code-Wise"?
Example 2:
Consider this config:
spring.cloud.stream.bindings.input.destination=orgChangeTopic
spring.cloud.stream.bindings.input.content-type=application/json
spring.cloud.stream.bindings.input.group=licensingGroup
spring.cloud.stream.kafka.binder.zkNodes=kafka
spring.cloud.stream.kafka.binder.brokers=kafka
And this code:
#SpringBootApplication
#RefreshScope
#EnableDiscoveryClient
#EnableFeignClients
#EnableEurekaClient
#EnableBinding(Sink.class)
public class LicenseServiceApplication {
public static void main(String[] args) {
SpringApplication.run(LicenseServiceApplication.class, args);
}
#StreamListener(Sink.INPUT)
public void loggerSink(OrganizationChangeModel orgChange) {
log.info("Received an {} event for organization id {}",
orgChange.getAction(), orgChange.getOrganizationId());
}
What this method is supposed to do is to fire whenever a message is fired in orgChangeTopic, we want the method loggerSink to fire.
How do I do this in Spring Boot 3?
In Spring Cloud Stream 4.0.0 (the version used if you are using Boot 3), a few things are removed - such as the EnableBinding, StreamListener, etc. We deprecated them before in 3.x and finally removed them in the 4.0.0 version. The annotation-based programming model is removed in favor of the functional programming style enabled through the Spring Cloud Function project. You essentially express your business logic as java.util.function.Funciton|Consumer|Supplier etc. for a processor, sink, and source, respectively. For ad-hoc source situations, as in your first example, Spring Cloud Stream provides a StreamBridge API for custom sends.
Your example #1 can be re-written like this:
#Component
public class SimpleSourceBean {
#Autowired
StreamBridge streamBridge
public void publishOrganizationChange(String action, String organizationId){
logger.debug("Sending Kafka message {} for Organization Id: {}", action, organizationId);
OrganizationChangeModel change = new OrganizationChangeModel(
OrganizationChangeModel.class.getTypeName(),
action,
organizationId,
UserContext.getCorrelationId());
streamBridge.send("output-out-0", MessageBuilder.withPayload(change).build());
}
}
Config
spring.cloud.stream.bindings.output-out-0.destination=orgChangeTopic
spring.cloud.stream.kafka.binder.brokers=kafka
Just so you know, you no longer need that zkNode property. Neither the content type since the framework auto-converts that for you.
StreamBridge send takes a binding name and the payload. The binding name can be anything - but for consistency reasons, we used output-out-0 here. Please read the reference docs for more context around the reasoning for this binding name.
If you have a simple source that runs on a timer, you can express this simply as a supplier as below (instead of using a StreamBrdige).
#Bean
public Supplier<OrganizationChangeModel> ouput() {
return () -> {
// return the payload
};
}
spring.cloud.function.definition=output
spring.cloud.bindings.output-out-0.destination=...
Example #2
#Bean
public Consumer<OrganizationChangeModel> loggerSink() {
return model -> {
log.info("Received an {} event for organization id {}",
orgChange.getAction(), orgChange.getOrganizationId());
};
}
Config:
spring.cloud.function.definition=loggerSink
spring.cloud.stream.bindings.loggerSink-in-0.destination=orgChangeTopic
spring.cloud.stream.bindings.loggerSinnk-in-0.group=licensingGroup
spring.cloud.stream.kafka.binder.brokers=kafka
If you want the input/output binding names to be specifically input or output rather than with in-0, out-0 etc., there are ways to make that happen. Details for this are in the reference docs.
I would like to use onExceptionProcessor to catch any exceptions catched by my route builder and save them in the database.
I don't khnow if i have to use onException(Exception.class) or errorHandler() and how to implement them correctly!
I tried the try-catch but it does not catch my exception (null pointer that i throw in the processor1). May be i do not implement it correctly ?
Here is my routeBuilder:
#component
public class MyRoute extends RouteBuilder {
#Autowired
private Processor processor1;
#Autowired
private Processor procssor2;
#Autowired
private Processor processor3;
#Autowired
private Processor onExceptionProcessor; // it a processor where i try to save the stacktrace of exception in the database
#Override
public void configure() throws Exception {
from(jmsDecoupageRouteIn)
.id("route_id_processing").messageHistory().transacted()
.log(LoggingLevel.DEBUG, log, "It's for just for log").pipeline()
.process(processor1)
.id(processor1.getClass().getSimpleName().toLowerCase())
.process(procssor2)
.id(procssor2.getClass().getSimpleName().toLowerCase())
.process(processor3)
.id(processor3.getClass().getSimpleName().toLowerCase())
.doTry()
.to(jmsDecoupageRouteOut)
.doCatch(Exception.class)
.log(LoggingLevel.ERROR, "EXCEPTION: ${exception.stacktrace}")
.process(onExceptionProcessor)
.id(onExceptionProcessor.getClass().getSimpleName().toLowerCase())
.endDoTry();
}
}
This is the generic structure of a doTry()...doCatch()...end() construct.
from("direct:start")
.doTry()
.process(new ProcessorFail())
.to("mock:result")
.doCatch(IOException.class, IllegalStateException.class)
.to("mock:catch")
.doFinally()
.to("mock:finally")
.end();
In your case you are using a .endDoTry() instead of .end(). Its a tiny gotcha in the Camel API. Change it and see if it works as expected.
Additional Reference
A test case for similar construct
Docs for doTry()...doCatch()...end()
Keep in mind that when you use doTry()...doCatch()...end() the regular Camel OnException handlers will not work (You can't mix them together).
Update: Screenshot as shared with OP
I am developing an application using JADE, JASON (Agent frameworks) and Spring Boot. Basically what I have is a JADE Container where Both Jade and Jason Agents are registered in. And Since I am using Spring, I tend to Autowire services. In that case I am in need to access some services, inside some of my Jason internal actions (which I custom wrote extending DefaultInternalAction class). which seems not working. I have the idea how to Autowire and how the Beans work. My doubt is whether those internal actions are in the spring context or not. I guess they are not. Thats why may be the Autowire thing is not working. Can someone please explain me about the real action inside the jade container and internal actions so that I can think differently about using Autowire inside jason internal actions.
As far as I know, internal actions is created by jason, not spring that is why you cant autowire services. Personnaly, I create factory and use it for getting instance of a service. Something like this:
public class SpringPluginFactory {
private static final SpringPluginFactory INSTANCE = new SpringPluginFactory();
private ApplicationContext applicationContext;
private SpringPluginFactory(){}
private <T> T createPlugin(Class<T> iface) {
if(applicationContext == null){
throw new IllegalStateException("applicationContext cannot be null");
}
try {
return applicationContext.getBean(iface);
} catch (Exception e) {
throw new RuntimeException("factory unable to construct instance of " + iface.getName());
}
}
public static <T> T getPlugin(Class<T> iface){
return INSTANCE.createPlugin(iface);
}
public void setApplicationContext(ApplicationContext applicationContext) {
this.applicationContext = applicationContext;
}
}
then I create bean in order to set aplicationContext:
#Bean
public SpringPluginFactory pluginFactory(ApplicationContext applicationContext){
SpringPluginFactory pluginFactory = SpringPluginFactory.INSTANCE;
pluginFactory.setApplicationContext(applicationContext);
return pluginFactory;
}
and use the factory in any behaviours or internal actions
SpringPluginFactory.getPlugin(YouService.class).doSomething();
Maybe it will help.
maybe someone has an idea to my following problem:
I am currently on a project, where i want to use the AWS SQS with Spring Cloud integration. For the receiver part i want to provide a API, where a user can register a "message handler" on a queue, which is an interface and will contain the user's business logic, e.g.
MyAwsSqsReceiver receiver = new MyAwsSqsReceiver();
receiver.register("a-queue-name", new MessageHandler(){
#Override
public void handle(String message){
//... business logic for the received message
}
});
I found examples, e.g.
https://codemason.me/2016/03/12/amazon-aws-sqs-with-spring-cloud/
and read the docu
http://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_sqs_support
But the only thing i found there to "connect" a functionality for processing a incoming message is a annotation on a method, e.g. #SqsListener or #MessageMapping.
These annotations are fixed to a certain queue-name, though. So now i am at a loss, how to dynamically "connect" my provided "MessageHandler" (from my API) to the incoming message for the specified queuename.
In the Config the example there is a SimpleMessageListenerContainer, which gets a QueueMessageHandler set, but this QueueMessageHandler does not seem
to be the right place to set my handler or to override its methods and provide my own subclass of QueueMessageHandler.
I already did something like this with the Spring Amqp integration and RabbitMq and thought, that it would be also similar here with AWS SQS.
Does anyone have an idea, how to accomplish this?
thx + bye,
Ximon
EDIT:
I found, that Spring JMS could actually do that, e.g. www.javacodegeeks.com/2016/02/aws-sqs-spring-jms-integration.html. Does anybody know, what consequences using JMS protocol has here, good or bad?
I am facing the same issue.
I am trying to go in an unusual way where I set up an Aws client bean at build time and then instead of using sqslistener annotation to consume from the specific queue I use the scheduled annotation which I can programmatically pool (each 10 secs in my case) from which queue I want to consume.
I did the example that iterates over queues defined in properties and then consumes from each one.
Client Bean:
#Bean
#Primary
public AmazonSQSAsync awsSqsClient() {
return AmazonSQSAsyncClientBuilder
.standard()
.withRegion(Regions.EU_WEST_1.getName())
.build();
}
Consumer:
// injected in the constructor
private final AmazonSQSAsync awsSqsClient;
#Scheduled(fixedDelay = 10000)
public void pool() {
properties.getSqsQueues()
.forEach(queue -> {
val receiveMessageRequest = new ReceiveMessageRequest(queue)
.withWaitTimeSeconds(10)
.withMaxNumberOfMessages(10);
// reading the messages
val result = awsSqsClient.receiveMessage(receiveMessageRequest);
val sqsMessages = result.getMessages();
log.info("Received Message on queue {}: message = {}", queue, sqsMessages.toString());
// deleting the messages
sqsMessages.forEach(message -> {
val deleteMessageRequest = new DeleteMessageRequest(queue, message.getReceiptHandle());
awsSqsClient.deleteMessage(deleteMessageRequest);
});
});
}
Just to clarify, in my case, I need multiple queues, one for each tenant, with the queue URL for each one passed in a property file. Of course, in your case, you could get the queue names from another source, maybe a ThreadLocal which has the queues you have created in runtime.
If you wish, you can also try the JMS approach where you create message consumers and add a listener to each one you wish (See the doc Aws Jms documentation).
When we do Spring and SQS we use the spring-cloud-starter-aws-messaging.
Then just create a Listener class
#Component
public class MyListener {
#SQSListener(value="myqueue")
public void listen(MyMessageType message) {
//process the message
}
}
Using Spring RestTemplate to invoke client rest calls, would it be possible to throttle these calls?
E.g. max 10 concurrent calls.
The RestTemplate itself does not seem to provide this itself so I wonder what the options are.
It would be best to have a generic solution to e.g. also throttle SOAP calls.
From the docs:
To create an instance of RestTemplate you can simply call the default
no-arg constructor. This will use standard Java classes from the
java.net package as the underlying implementation to create HTTP
requests. This can be overridden by specifying an implementation of
ClientHttpRequestFactory. Spring provides the implementation
HttpComponentsClientHttpRequestFactory that uses the Apache
HttpComponents HttpClient to create requests.
HttpComponentsClientHttpRequestFactory is configured using an instance
of org.apache.http.client.HttpClient which can in turn be configured
with credentials information or connection pooling functionality.
I'd look into configuring RestTemplate to use HTTP Components and play with setMaxPerRoute and setMaxTotal. If your SOAP client also happens to be using HTTP Components there may be a way to share the Commons HTTP Components settings between the two.
The other option is to roll your own. You could create a Proxy that uses a Semaphore to block until another request is finished. Something along these lines (note that this code is totally untested and is only to communicate the general idea of how you'd implement this):
public class GenericCounterProxy implements InvocationHandler
{
private final Object target;
private final int maxConcurrent;
private final Semaphore sem;
GenericCounterProxy(Object target, int maxConcurrent)
{
this.target = target;
this.maxConcurrent = maxConcurrent;
this.sem = new Semaphore(maxConcurrent, true);
}
#Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
try
{
// block until acquire succeeds
sem.acquire()
method.invoke(target, args);
}
finally
{
// release the Semaphore no matter what.
sem.release();
}
}
public static <T> T proxy(T target, int maxConcurrent)
{
InvocationHandler handler = new GenericCounterProxy(target, maxConcurrent);
return (T) Proxy.newProxyInstance(
target.getClass().getClassLoader(),
target.getClass().getInterfaces(),
handler);
}
}
If you wanted to go with this type of approach:
You should probably refine the methods for which the proxy acquires the Semaphore since not every method on the target would be subject to throttling (for example, getters for settings).
You need to change from RestTemplate to RestOperations which is an interface or change the proxying mechanism to use class based proxying.