Apache Camel-Master/Slave - spring-boot

I've spring boot application which has a set of camel routes. I'm looking for an option to achieve failover in camel routes when one JVM goes down. My goal is to have my app running in one JVM and when that application goes down, another JVM's route should get process my messages.
When I try to add the Clustering, I'm getting an error (Caused by: java.lang.IllegalStateException: CamelCluster service not found) and even I'm not sure whether the way I'm trying my code is correct.
public class RouteCmdLineRunner implements CommandLineRunner {
#Autowired
private Configuration configuration;
#Autowired
private CamelContext camelContext;
#Override
public void run(String... args) {
CamelClusterService atomixClusterService = new AtomixClusterService();
atomixClusterService.setId("camel-node-1");
camelContext.addService(atomixClusterService);
if (configuration != null && configuration.getRoutes() != null) {
configuration.getRoutes().forEach(route -> {
try {
camelContext.addRoutePolicyFactory(ClusteredRoutePolicyFactory.forNamespace("my-ns"));
camelContext.addRoutes(new MyRouteBuilder(route, configuration));
} catch (Exception e) {
throw new RuntimeException(e);
}
});
}
}
}
application.yml
camel:
component:
atomix:
cluster:
service:
id: testid-1
enabled: true
order: 1
mode: node
address: localhost:8081
master:
service: AtomixClusterService
camel.clustered.controller.namespace: my-ns
camel.clustered.controller.enabled: true
camel.component.master.service: true
pom.xml
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-master-starter</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-atomix-starter</artifactId>
<version>3.0.0-RC3</version>
</dependency>
<dependency>
<groupId>io.atomix</groupId>
<artifactId>atomix-all</artifactId>
<version>1.0.8</version>
</dependency>
Is Clustering in Camel in the experimental stage? https://camel.apache.org/manual/latest/clustering.html
Camel documentation says it has a master component which does the failover. (https://camel.apache.org/components/latest/master-component.html) but I don't see the complete example for Clustering.
What is the use of a camel clustered controller?
Although there is camel documentation still it is not complete and confusing a lot.
Camel Version: 3.1.0
Spring-boot: 2.2.5.RELEASE
Any pointers would be helpful in achieving Camel clustering. Am I missing anything conceptually?.
For this failover, I don't have the option to install any new server like ZooKeeper/Consul servers.

There is an example on the camel-spring-boot example repo: https://github.com/apache/camel-spring-boot/tree/master/examples/camel-example-spring-boot-clustered-route-controller
Looking at your code I see a number of issues:
you do configure a cluster service programmatically in your run method as well as using spring boot properties
the configuration of master component set AtomixClusterService as service to use but it does not look like you have a bean
so named
One of the two is probably the the reason you are hitting IllegalStateException.
About your questions:
Yes it is still experimental
There is an example at the end of the clustering doc (which is definitively not completed)
the clustered controller is in charge to start/stop routes when the leadership is taken/lost automatically without having you to add any route policy or configuring master component.

Related

I am not able to connect to Google Cloud Memory Store from Spring Boot

I am developing a module with spring boot in my backend where i need to use Redis through GCP Memory Store. I have been searching in forum and even the "oficial documentation" about memory store but i cannot understand how to connect to memory store with my spring boot app.
I found a google code lab but they use a Compute Engine VM to install spring boot and then save and retrieve information from memory store. So i tried to do it like that in my local spring boot but it didnt work because throws an error saying:
Unable to connect to Redis; nested exception is io.lettuce.core.RedisConnectionException: Unable to connect to 10.1.3.4
the codelab i mentioned earlier says that you only have to add this line to your application.properties:
spring.redis.host=10.1.3.4
as well as the annotation #EnableCaching in the main class and #Cachable annotation in the controller method where you try to do something with redis.
the method looks like this:
#RequestMapping("/hello/{name}")
#Cacheable("hello")
public String hello(#PathVariable String name) throws InterruptedException {
Thread.sleep(5000);
return "Hello " + name;
}
i dont know what else to do. Notice that i am new on this topic of redis and memory store.
Anyone can give me some guidance on this please?
thanks in advance
codelab url: https://codelabs.developers.google.com/codelabs/cloud-spring-cache-memorystore#0
See this documentation on how to setup Memorystore Redis instance.
Included in the documentation is how you can connect and test your Memorystore instance from different computing environments.
There's also a step by step guide on how SpringBoot can use Redis to cache with annonations.
Add the Spring Data Redis starter in your pom.xml if you're using Maven for your project setup.
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-boot-starter-cache</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
Add this configuration in your application.properties file:
spring.redis.host=<MEMORYSTORE_REDIS_IP>
# Configure default TTL, e.g., 10 minutes
spring.cache.redis.time-to-live=600000
Turn on caching capability explicitly with the #EnableCaching annotation:
#SpringBootApplication
#EnableCaching
class DemoApplication {
...
}
Once you configured the Spring Boot with Redis and enabled caching, you can use the #Cacheable annotation to cache return values.
#Service
class OrderService {
private final OrderRepository orderRepository;
public OrderService(OrderRepository orderRepository) {
this.orderRepository = orderRepository;
}
#Cacheable("order")
public Order getOrder(Long id) {
orderRepository.findById(id);
}
}

Spring cloud bus with AWS Kinesis stream #refreshscope

I read everywhere #RefreshScope for cloud bus applications work with RabbitMQ and Kafka. But in my case, I am using AWS Parameter store. I want all my client instances to be refreshed automatically without rebuilding servers on AWS Console.
I created AWS Eventbridge from Paramstore to notify Kinesis Stream but I am not able to figure out how can it notify all my client nodes instead of load balancer refresh to only one node(instance).
Thank you for responding in advance.
I've never worked with AWS Eventbridge / Kinesis, however:
#RefreshScope is something that belongs to spring cloud and not not AWS.
More precisely, beans defined with this scope will be re-loaded by spring without reloading the whole application context "dynamically" when configuration changes in spring boot cloud configuration service. Usually this means that you don't have to restart the application.
Now, spring boot microservice should be deployed with actuator that exposes refresh endpoint. Calling this endpoint manually will cause all the #RefreshScope beans to reload.
Here is the source code of the RefreshEndpoint:
#Endpoint(id = "refresh")
public class RefreshEndpoint {
private ContextRefresher contextRefresher;
public RefreshEndpoint(ContextRefresher contextRefresher) {
this.contextRefresher = contextRefresher;
}
#WriteOperation
public Collection<String> refresh() {
Set<String> keys = this.contextRefresher.refresh();
return keys;
}
}
As you see, its merely invokes contextRefresher.refresh(), ContextRefresher is a bean that you can inject in your custom code that will listen to the changes coming from AWS Parameter store (it should somehow invoke it directly or maybe send some message that you could consume or something).
If you're using spring-cloud-bus (disclaimer, I've never worked with it) it exposes the bus-refresh endpoint as well (pretty similar mechanism to what I've described), read spring-cloud-bus documentation for more details.
Thank you Team for sharing info.
Here is what I did to make it work. Added these two libraries to my project
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kinesis</artifactId>
<version>1.1.0.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-bus</artifactId>
</dependency>
And added these two entries into bootstrap.properties
cloud.aws.region.static=us-east-1
cloud.aws.stack.auto = false
And refreshing using this endpoint (/bus-refresh)

Metrics http_server_requests_seconds_count in spring boot application using cxf-spring-boot-starter-jaxrs contains uri as "UNKNOWN"

Metrics http_server_requests_seconds_count in Spring Boot application with version 2.0.8.Release exposed using spring actuator contains URI as:
"UNKNOWN".
Spring Boot application is using cxf-spring-boot-starter-jaxrs for exposing rest endpoints.
I have added micrometer-registry-prometheus dependency in my project.
http_server_requests_seconds_count{exception="None",method="POST",status="200",uri="UNKNOWN",} 2.0
I have tried adding micrometer-jersey2 dependency in my project.
Actual
http_server_requests_seconds_count{exception="None",method="POST",status="200",uri="UNKNOWN",} 2.0
Expected:
http_server_requests_seconds_count{exception="None",method="GET",status="200",uri="/sayHello",} 2.0
After the clarification in OP comments (CXF being another JAX-RS implementation): There's currently no support in Micrometer to handle CXF requests. It (Spring WebMvc) can't extract the optionally parameterized request url and in that case falls back to UNKNOWN. (Otherwise this could lead to a metrics explosion if your CXF endpoints provide some highly parameterizable URLs which get a lot of traffic.)
So you could have a look at the micrometer-jersey2 implementation and derive a micrometer-cxf implementation ;) (Or if not already the case (use the search) - open up an issue with the Micrometer or CXF project. I am mentioning the latter, because they might be interessted in taking care of that implementation.)
If you need cxf statistics for micrometer report, you can try
<dependency>
<groupId>io.github.kdprog</groupId>
<artifactId>cxf-micrometer-metrics</artifactId>
<version>1.0.0</version>
</dependency>
The following statistics will be reported
cxf_requests_processed_total - total number of cxf requests ,
cxf_requests_seconds_sum - total execution time of cxf requests,
cxf_requests_seconds_max - maximum execution time of cxf request,
cxf_requests_success_total - total number of successfully processed cxf requests,
cxf_requests_failed_total - total number of failed cxf requests
for each web service method of every client or server cxf endpoint.
For spring applications add the following bean to your application configuration.
#Bean
public FactoryBeanListener cxfMicrometerBean(final MeterRegistry registry) {
return new MicrometerFactoryBeanListener(registry);
}
https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.supported
You can make custom tag provider to override it:
#Bean
WebMvcTagsProvider webMvcTagsProvider() {
return new DefaultWebMvcTagsProvider() {
#Override
public Iterable<Tag> getTags(HttpServletRequest request, HttpServletResponse response,
Object handler, Throwable exception) {
return Tags.concat(
super.getTags(request, response, handler, exception),
Tags.of(Tag.of("uri",request.getRequestURI()))
);
}
};
}
More examples.
You can also collect cxf metrics for prometheus using io.github.ddk-prog:cxf-prometheus-metrics.
Add dependency to your pom.xml
<dependency>
<groupId>io.github.ddk-prog</groupId>
<artifactId>cxf-prometheus-metrics</artifactId>
<version>1.0.0</version>
</dependency>
and the following bean to your application configuration.
#Bean
public FactoryBeanListener cxfPrometheusFeatureBean(final CollectorRegistry registry) {
return new PrometheusFactoryBeanListener(registry);
}
You will get cxf_requests_total, cxf_requests_success, cxf_requests_failed, cxf_requests_seconds for each endpoint and operation in your spring boot actuator prometheus report.
For example,
cxf_requests_seconds{endpoint="server1",operation="server1Method",} 0.0157349
If you are using WebFlux on your project, you can make your custom tag provider by overriding:
#Bean
WebFluxTagsProvider webFluxTagsProvider() {
return new DefaultWebFluxTagsProvider() {
#Override
public Iterable<Tag> httpRequestTags(ServerWebExchange exchange, Throwable exception) {
return Tags.concat(super.httpRequestTags(exchange, exception), Tags.of(Tag.of("uri", exchange.getRequest().getPath().value())));
}
};
}
It works for me.

Spring boot Artemis embedded broker behaviour

Morning all,
I've been struggling lately with the spring-boot-artemis-starter.
My understanding of its spring-boot support was the following:
set spring.artemis.mode=embedded and, like tomcat, spring-boot will instanciate a broker reachable through tcp (server mode). The following command should be successful: nc -zv localhost 61616
set spring.artmis.mode=native and spring-boot will only configure the jms template according to the spring.artemis.* properties (client mode).
The client mode works just fine with a standalone artemis server on my machine.
Unfortunatelly, I could never manage to reach the tcp port in server mode.
I would be grateful if somebody confirms my understanding of the embedded mode.
Thank you for tour help
After some digging I noted that the implementation provided out of the box by the spring-boot-starter-artemis uses org.apache.activemq.artemis.core.remoting.impl.invm.InVMAcceptorFactory acceptor. I'm wondering if that's not the root cause (again I'm by no means an expert).
But it appears that there is a way to customize artemis configuration.
Therefore I tried the following configuration without any luck:
#SpringBootApplication
public class MyBroker {
public static void main(String[] args) throws Exception {
SpringApplication.run(MyBroker.class, args);
}
#Autowired
private ArtemisProperties artemisProperties;
#Bean
public ArtemisConfigurationCustomizer artemisConfigurationCustomizer() {
return configuration -> {
try {
configuration.addAcceptorConfiguration("netty", "tcp://localhost:" + artemisProperties.getPort());
} catch (Exception e) {
throw new RuntimeException("Failed to add netty transport acceptor to artemis instance");
}
};
}
}
You just have to add a Connector and an Acceptor to your Artemis Configuration. With Spring Boot Artemis starter Spring creates a Configuration bean which will be used for EmbeddedJMS configuration. You can see this in ArtemisEmbeddedConfigurationFactory class where an InVMAcceptorFactory will be set for the configuration. You can edit this bean and change Artemis behaviour through custom ArtemisConfigurationCustomizer bean which will be sucked up by Spring autoconfig and be applied to the Configuration.
An example config class for your Spring Boot application:
import org.apache.activemq.artemis.api.core.TransportConfiguration;
import org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptorFactory;
import org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnectorFactory;
import org.springframework.boot.autoconfigure.jms.artemis.ArtemisConfigurationCustomizer;
import org.springframework.context.annotation.Configuration;
#Configuration
public class ArtemisConfig implements ArtemisConfigurationCustomizer {
#Override
public void customize(org.apache.activemq.artemis.core.config.Configuration configuration) {
configuration.addConnectorConfiguration("nettyConnector", new TransportConfiguration(NettyConnectorFactory.class.getName()));
configuration.addAcceptorConfiguration(new TransportConfiguration(NettyAcceptorFactory.class.getName()));
}
}
My coworker and I had the exact same problem as the documentation on this link (chapter Artemis Support) says nothing about adding an individual ArtemisConfigurationCustomizer - Which is sad because we realized that without this Customizer our Spring Boot App would start and act as if everything was okay but actually it wouldn't do anything.
We also realized that without the Customizer the application.properties file is not beeing loaded so no matter what host or port you mentioned there it would not count.
After adding the Customizer as stated by the two examples it worked without a problem.
Here some results that we figured out:
It only loaded the application.properties after configuring an ArtemisConfigurationCustomizer
You don't need the broker.xml anymore with an embedded spring boot artemis client
Many examples showing the use of Artemis use a "in-vm" protocol while we just wanted to use the netty tcp protocol so we needed to add it into the configuration
For me the most important parameter was pub-sub-domain as I was using topics and not queues. If you are using topics this parameter needs to be set to true or the JMSListener won't read the messages.
See this page: stackoverflow jmslistener-usage-for-publish-subscribe-topic
When using a #JmsListener it uses a DefaultMessageListenerContainer
which extends JmsDestinationAccessor which by default has the
pubSubDomain set to false. When this property is false it is
operating on a queue. If you want to use topics you have to set this
properties value to true.
In Application.properties:
spring.jms.pub-sub-domain=true
If anyone is interested in the full example I have uploaded it to my github:
https://github.com/CorDharel/SpringBootArtemisServerExample
The embedded mode starts the broker as part of your application. There is no network protocol available with such setup, only InVM calls are allowed. The auto-configuration exposes the necessary pieces you can tune though I am not sure you can actually have a TCP/IP channel with the embedded mode.

Configuring the AJP port on Jetty in a Spring boot application

I am trying to configure the AJP port for the Jetty server in my spring boot application. However, I've seen examples about the Tomcat AJP connector but not for Jetty. Can someone tell me how I should configure the AJP port on Jetty?
Spring boot bundles Jetty 9.3 by default. The AJP feature has been droped in jetty 9. If you really need to use AJP, then you will have to add jetty 8 to your classpath, and write a custom JettyServerCustomizer (http://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/context/embedded/jetty/JettyServerCustomizer.html).
I don't think you will find anything in spring documentation regarding this configuration. Once you get to write your customizer, you'll have to read through Jetty documentation to find out how to achieve your AJP configuration.
Jetty has droped AJP in favor of HTTP connector. Unless you have a real use case, I would avise to migrate to HTTP.
I haven't tried the code, but this should work (needs to be wrapped in spring boot code)
return new JettyServerCustomizer() {
#Override
public void customize(Server server) {
s.addConnector(new Ajp13SocketConnector());
}
};
You need jetty-server 8.x and jetty-ajp 8.x in your classpath. With latest 8.x releases, your pom.xml (if you're a maven user) should contain:
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<version>8.1.19.v20160209</version>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-ajp</artifactId>
<version>8.1.19.v20160209</version>
</dependency>
Full Sample of The Example Suggested Above For anyone's reference
#Bean
public EmbeddedServletContainerFactory jettyContainer() {
JettyEmbeddedServletContainerFactory factory = new JettyEmbeddedServletContainerFactory();
JettyServerCustomizer customizers = new JettyServerCustomizer() {
#Override
public void customize(Server server) {
// Connector connector = new Ajp13SocketConnector();
// connector.setPort(9009); //if you want a custom port
server.addConnector(new Ajp13SocketConnector());
}
};
factory.addServerCustomizers(customizers);
return factory;
}
Logs
o.e.jetty.server.AbstractConnector: Started
SelectChannelConnector#0.0.0.0:7090 o.e.jetty.server.AbstractConnector
: Started Ajp13SocketConnector#0.0.0.0:9009
o.e.jetty.ajp.Ajp13SocketConnector : AJP13 is not a secure protocol.
Please protect port 9009 .s.b.c.e.j.JettyEmbeddedServletContainer :
Jetty started on port(s) 7090, 9009

Resources