I have two projects one the service project another one consumer project,
Consumer project consumes the services of other project and the call should be async using JMS
I installed jms plugin in both of the projects
I have defined the JMSConnectionFactory in both of the project as below in resources.groovy
import org.springframework.jms.connection.SingleConnectionFactory
import org.apache.activemq.ActiveMQConnectionFactory
beans = {
jmsConnectionFactory(org.apache.activemq.ActiveMQConnectionFactory) { brokerURL = 'vm://localhost' }
}
Note: Both of the project are for now on same machine (i.e. localhost)
Now from consumer's controller I am making call to service from ServiceProvider project
jmsService.send(service:'serviceProvider', params.body)
In ServiceProvider the service is defined as follow
import grails.plugin.jms.*
class ServiceProviderService {
def jmsService
static transactional = true
static exposes = ['jms1']
def createMessage(msg) {
print "Called1"
sleep(2000) // slow it down
return null
}
}
now when controller submits the call to service it gets submitted successfully but doesn't reach to the actual service
I also tried
jmsService.send(app: "ServiceProvider", service: "serviceProvider", method: "createMessage", msg, "standard", null)
Update
Now I have installed activeMQ plugin to service provider to make it embedded broker (jms is already there)
and created a service
package serviceprovider
class HelloService {
boolean transactional = false
static exposes = ['jms']
static destination = "queue.notification"
def onMessage(it){
println "GOT MESSAGE: $it"
}
def sayHello(String message){
println "hello"+message
}
}
resources.groovy in both of the project is now
import org.springframework.jms.connection.SingleConnectionFactory
import org.apache.activemq.ActiveMQConnectionFactory
beans = {
jmsConnectionFactory(org.apache.activemq.ActiveMQConnectionFactory) { brokerURL = 'tcp://127.0.0.1:61616' }
}
from consumer's controller I am calling this service like below
jmsService.send(app:'queue.notification',service:'hello',method: 'sayHello', params.body)
call to method gets submitted but actually it is not getting called!
The in vm activemq config (vm://localhost) works only within a single VM. If your 2 projects run in separate VMs try setting up an external AMQ broker.
if you are using separate processes, then you need to use a different transport than VM (its for a single VM only), also, is one of your processes starting a broker? If not, then one of them should embed the broker (or run it externally) and expose it over a transport (like TCP)...
Related
We have Spring Boot services running in Kubernetes and are using the Spring Cloud Kubernetes Load Balancer functionality with RestTemplate to make calls to other Spring Boot services. One of the main reasons we have this in place is historical - in that previously we ran our services in EC2 using Eureka for service discovery and after the migration we kept the Spring discovery client/client-side load balancing in place (updating dependencies etc for it to work with the Spring Cloud Kubernetes project)
We have a problem that when one of the target pods goes down we get multiple failures for requests for a period of time with java.net.NoRouteToHostException ie the spring load balancer is still trying to send to that pod.
So I have a few questions on this:
Shouldn't the target instance get removed automatically when this happens? So it might happen once but after that, the target pod list will be repaired?
Or if not is there some other configuration we need to add to handle this - eg retry / circuit breaker, etc?
A more general question is what benefit does Spring's client-side load balancing bring with Kubernetes? Without it, our service would still be able to call other services using Kubernetes built-in service / load-balancing functionality and this should handle the issue of pods going down automatically. The Spring documentation also talks about being able to switch from POD mode to SERVICE mode (https://docs.spring.io/spring-cloud-kubernetes/docs/current/reference/html/index.html#loadbalancer-for-kubernetes). But isn't this service mode just what Kubernetes does automatically? I'm wondering if the simplest solution here isn't to remove the Spring Load Balancer altogether? What would we lose then?
An update on this: we had the spring-retry dependency in place, but the retry was not working as by default it only works for GETs and most of our calls are POST (but OK to call again). Adding the configuration spring.cloud.loadbalancer.retry.retryOnAllOperations: true fixed this, and hence most of these failures should be avoided by the retry using an alternative instance on the second attempt.
We have also added a RetryListener that clears the load balancer cache for the service on certain connection exceptions:
#Configuration
public class RetryConfig {
private static final Logger logger = LoggerFactory.getLogger(RetryConfig.class);
// Need to use bean factory here as can't autowire LoadBalancerCacheManager -
// - it's set to 'autowireCandidate = false' in LoadBalancerCacheAutoConfiguration
#Autowired
private BeanFactory beanFactory;
#Bean
public CacheClearingLoadBalancedRetryFactory cacheClearingLoadBalancedRetryFactory(ReactiveLoadBalancer.Factory<ServiceInstance> loadBalancerFactory) {
return new CacheClearingLoadBalancedRetryFactory(loadBalancerFactory);
}
// Extension of the default bean that defines a retry listener
public class CacheClearingLoadBalancedRetryFactory extends BlockingLoadBalancedRetryFactory {
public CacheClearingLoadBalancedRetryFactory(ReactiveLoadBalancer.Factory<ServiceInstance> loadBalancerFactory) {
super(loadBalancerFactory);
}
#Override
public RetryListener[] createRetryListeners(String service) {
RetryListener cacheClearingRetryListener = new RetryListener() {
#Override
public <T, E extends Throwable> boolean open(RetryContext context, RetryCallback<T, E> callback) { return true; }
#Override
public <T, E extends Throwable> void close(RetryContext context, RetryCallback<T, E> callback, Throwable throwable) {}
#Override
public <T, E extends Throwable> void onError(RetryContext context, RetryCallback<T, E> callback, Throwable throwable) {
logger.warn("Retry for service {} picked up exception: context {}, throwable class {}", service, context, throwable.getClass());
if (throwable instanceof ConnectTimeoutException || throwable instanceof NoRouteToHostException) {
try {
LoadBalancerCacheManager loadBalancerCacheManager = beanFactory.getBean(LoadBalancerCacheManager.class);
Cache loadBalancerCache = loadBalancerCacheManager.getCache(CachingServiceInstanceListSupplier.SERVICE_INSTANCE_CACHE_NAME);
if (loadBalancerCache != null) {
boolean result = loadBalancerCache.evictIfPresent(service);
logger.warn("Load Balancer Cache evictIfPresent result for service {} is {}", service, result);
}
} catch(Exception e) {
logger.error("Failed to clear load balancer cache", e);
}
}
}
};
return new RetryListener[] { cacheClearingRetryListener };
}
}
}
Are there any issues with this approach? Could something like this be added to the built in functionality?
Shouldn't the target instance get removed automatically when this
happens? So it might happen once but after that the target pod list
will be repaired?
To resolve this issue you have to use the Readiness and Liveness Probe in Kubernetes.
Readiness will check the health of the endpoint that your application has, on the period of interval. If the application fails it will mark your PODs as Unready to accept the Traffic. So no traffic will go to that POD(replica).
Liveness will restart your application if it fails so your container or we can say POD will come up again and once we will get 200 response from app K8s will mark your POD as Ready to accept the traffic.
You can create the simple endpoint in the application that give response as 200 or 204 as per need.
Read more at : https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Make sure you application using the Kubernetes service to talk with each other.
Application 1 > Kubernetes service of App 2 > Application 2 PODs
To enable load balancing based on Kubernetes Service name use the
following property. Then load balancer would try to call application
using address, for example service-a.default.svc.cluster.local
spring.cloud.kubernetes.loadbalancer.mode=SERVICE
The most typical way to use Spring Cloud LoadBalancer on Kubernetes is
with service discovery. If you have any DiscoveryClient on your
classpath, the default Spring Cloud LoadBalancer configuration uses it
to check for service instances. As a result, it only chooses from
instances that are up and running. All that is needed is to annotate
your Spring Boot application with #EnableDiscoveryClientto enable
K8s-native Service Discovery.
References : https://stackoverflow.com/a/68536834/5525824
I'm using Kafka embedded broker with spring boot and junit 5.I have been able to wire up successfully and see that the embedded broker is running.
In my setup method I pump in a few messages to the queue that my actual code listens on
#BeforeAll
public void setup() {
// code to play down some messages to topic X
}
My consumer/listener is never trigerred despite there being no errors encountered in the setup method
My Consumer is setup like
class Consumer() {
#KafkaListener(topics="X",
groupId ="...",
containerFactory="my-container-factory"
)
public void consume(ConsumerRecord<String,byte[] rec) {
//logic to handle
logger.info("Print rec : "+rec)
}
}
else where I've set up my ListenerContainerFactory with a name like
#Bean(name="my-container-factory")
public KafkaContainerListenerFactory<String,byte[]> factory() {
}
What could be wrong with this?My assertions in the test case fail and additionally I don't see my log statements that should be printed if my consume method were ever called.
I've a feeling,that auto configuration due to #SpringBootTest and #EmbeddedKafka is setting up some other listener container factory and so maybe my #KafkaListener annotation is wrong.
I know,its a bit vague but could you please tell me what/where to look at?If I run as a #SpringBootApplication my Consumer is pulling in messages from the actual queue.So no problems with my actual app.Its the test that's not executing as per expectation.
Please help.
Edit 1:
I have spring.kafka.consumer.auto-offset-reset=earliest set in my yml file.
I am having existing Spring Boot application and I want to do monitoring the application through actuator.I tried with http endpoints and it is working fine for me. Instead of http end points I need JMX end points for my existing running application.
If you add spring-boot-starter-actuatordependency in your build.gradle or pom.xml file you will have JMX bean enabled by default as well as HTTP Endpoints.
You can use JConsole in order to view your JMX exposed beans. You'll find more info about this here.
More details about how to access JMX endpoints here.
Assuming you're using a Docker image where the entry point is the Spring Boot app using java in which case the PID is "1" and so would the Attach API's Virtual Machine ID. You can implement a health probe as follows.
import com.sun.tools.attach.spi.AttachProvider;
import java.util.Map;
import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
public class HealthProbe {
public static void main(String[] args) throws Exception {
final var attachProvider = AttachProvider.providers().get(0);
final var virtualMachine = attachProvider.attachVirtualMachine("1");
final var jmxServiceUrl = virtualMachine.startLocalManagementAgent();
try (final var jmxConnection = JMXConnectorFactory.connect(new JMXServiceURL(jmxServiceUrl))) {
final MBeanServerConnection serverConnection = jmxConnection.getMBeanServerConnection();
#SuppressWarnings("unchecked")
final var healthResult =
(Map<String, ?>)
serverConnection.invoke(
new ObjectName("org.springframework.boot:type=Endpoint,name=Health"),
"health",
new Object[0],
new String[0]);
if ("UP".equals(healthResult.get("status"))) {
System.exit(0);
} else {
System.exit(1);
}
}
}
}
This will use the Attach API and make the original process start a local management agent.
The org.springframework.boot:type=Endpoint,name=Health object instance would have it's health method invoked which will provide a Map version of the /actuator/health output. From there the value of status should be UP if things are ok.
Then exit with 0 if ok, or 1 otherwise.
This can be embedded in an existing Spring Boot app so long as loader.main is set. The following is the HEALTHCHECK probe I used
HEALTHCHECK --interval=5s --start-period=60s \
CMD ["java", \
"-Dloader.main=net.trajano.swarm.gateway.healthcheck.HealthProbe", \
"org.springframework.boot.loader.PropertiesLauncher" ]
This is the technique I used in distroless Docker Image.
Side note: Don't try to put this in a CommandLineRunner interface because it will try to pull the configuration from the main app and you likely won't need the whole web stack.
maybe someone has an idea to my following problem:
I am currently on a project, where i want to use the AWS SQS with Spring Cloud integration. For the receiver part i want to provide a API, where a user can register a "message handler" on a queue, which is an interface and will contain the user's business logic, e.g.
MyAwsSqsReceiver receiver = new MyAwsSqsReceiver();
receiver.register("a-queue-name", new MessageHandler(){
#Override
public void handle(String message){
//... business logic for the received message
}
});
I found examples, e.g.
https://codemason.me/2016/03/12/amazon-aws-sqs-with-spring-cloud/
and read the docu
http://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_sqs_support
But the only thing i found there to "connect" a functionality for processing a incoming message is a annotation on a method, e.g. #SqsListener or #MessageMapping.
These annotations are fixed to a certain queue-name, though. So now i am at a loss, how to dynamically "connect" my provided "MessageHandler" (from my API) to the incoming message for the specified queuename.
In the Config the example there is a SimpleMessageListenerContainer, which gets a QueueMessageHandler set, but this QueueMessageHandler does not seem
to be the right place to set my handler or to override its methods and provide my own subclass of QueueMessageHandler.
I already did something like this with the Spring Amqp integration and RabbitMq and thought, that it would be also similar here with AWS SQS.
Does anyone have an idea, how to accomplish this?
thx + bye,
Ximon
EDIT:
I found, that Spring JMS could actually do that, e.g. www.javacodegeeks.com/2016/02/aws-sqs-spring-jms-integration.html. Does anybody know, what consequences using JMS protocol has here, good or bad?
I am facing the same issue.
I am trying to go in an unusual way where I set up an Aws client bean at build time and then instead of using sqslistener annotation to consume from the specific queue I use the scheduled annotation which I can programmatically pool (each 10 secs in my case) from which queue I want to consume.
I did the example that iterates over queues defined in properties and then consumes from each one.
Client Bean:
#Bean
#Primary
public AmazonSQSAsync awsSqsClient() {
return AmazonSQSAsyncClientBuilder
.standard()
.withRegion(Regions.EU_WEST_1.getName())
.build();
}
Consumer:
// injected in the constructor
private final AmazonSQSAsync awsSqsClient;
#Scheduled(fixedDelay = 10000)
public void pool() {
properties.getSqsQueues()
.forEach(queue -> {
val receiveMessageRequest = new ReceiveMessageRequest(queue)
.withWaitTimeSeconds(10)
.withMaxNumberOfMessages(10);
// reading the messages
val result = awsSqsClient.receiveMessage(receiveMessageRequest);
val sqsMessages = result.getMessages();
log.info("Received Message on queue {}: message = {}", queue, sqsMessages.toString());
// deleting the messages
sqsMessages.forEach(message -> {
val deleteMessageRequest = new DeleteMessageRequest(queue, message.getReceiptHandle());
awsSqsClient.deleteMessage(deleteMessageRequest);
});
});
}
Just to clarify, in my case, I need multiple queues, one for each tenant, with the queue URL for each one passed in a property file. Of course, in your case, you could get the queue names from another source, maybe a ThreadLocal which has the queues you have created in runtime.
If you wish, you can also try the JMS approach where you create message consumers and add a listener to each one you wish (See the doc Aws Jms documentation).
When we do Spring and SQS we use the spring-cloud-starter-aws-messaging.
Then just create a Listener class
#Component
public class MyListener {
#SQSListener(value="myqueue")
public void listen(MyMessageType message) {
//process the message
}
}
I have a system where external systems can subscribe to events generated by my system. The system is written in Grails 2, using the RabbitMQ plugin for internal messaging. The events to external systems are communicated via HTTP.
I would like to create a queue for each subscriber to prevent that a slow subscriber endpoint slows down messages to an other subscriber. Subscriptions can occur runtime, that's why defining the queues in the application config is not desirable.
How can I create a queue with a topic binding runtime with the Grails RabbitMQ plugin?
As reading messages from RabbitMQ queues is directly coupled to services, a side-problem to creating the queue runtime could be to have multiple instances of that Grails service. Any ideas?
I don't have a ready solution for You but if you follow the code in the RabbitmqGrailsPlugin Descriptor especially the doWithSpring section
You should be able to recreate the steps necessary to initialize a new Queue and associated Listener dynamically at runtime.
It all comes down then to pass the needed parameters, register necessary spring beans and start the listeners.
To answer your second question I think you can come up with some naming convention and create a new queue handler for each queue. An example how to create spring beans dynamically can be found here: dynamically declare beans
Just a short example how I would quickly register a Queue it requires much more wiring etc...
def createQ(queueName) {
def queuesConfig = {
"${queueName}"(durable: true, autoDelete: false,)
}
def queueBuilder = new RabbitQueueBuilder()
queuesConfig.delegate = queueBuilder
queuesConfig.resolveStrategy = Closure.DELEGATE_FIRST
queuesConfig()
queueBuilder.queues?.each { queue ->
if (log.debugEnabled) {
log.debug "Registering queue '${queue.name}'"
}
BeanDefinitionBuilder builder = BeanDefinitionBuilder.rootBeanDefinition(Queue.class);
builder.addConstructorArgValue(queue.name)
builder.addConstructorArgValue(Boolean.valueOf(queue.durable))
builder.addConstructorArgValue(Boolean.valueOf(queue.exclusive))
builder.addConstructorArgValue(Boolean.valueOf(queue.autoDelete))
builder.addConstructorArgValue(queue.arguments)
DefaultListableBeanFactory factory = (DefaultListableBeanFactory) grailsApplication.mainContext.getBeanFactory();
factory.registerBeanDefinition("grails.rabbit.queue.${queue.name}", builder.getBeanDefinition());
}
}
I ended up using Spring AMQP which is used by the Grails RabbitMQ plugin. Removed some methods/arguments as they are not relevant to the sample:
class MyUpdater {
void handleMessage(Object message) {
String content = new String(message)
// do whatever you need with the message
}
}
import org.springframework.amqp.core.BindingBuilder
import org.springframework.amqp.core.Queue
import org.springframework.amqp.core.TopicExchange
import org.springframework.amqp.rabbit.core.RabbitAdmin
import org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer
import org.springframework.amqp.rabbit.listener.adapter.MessageListenerAdapter
import org.springframework.amqp.support.converter.SimpleMessageConverter
import org.springframework.amqp.rabbit.connection.ConnectionFactory
class ListenerInitiator {
// autowired
ConnectionFactory rabbitMQConnectionFactory
protected void initiateListener() {
RabbitAdmin admin = new RabbitAdmin(rabbitMQConnectionFactory)
// normally passed to this method, moved to local vars for simplicity
String queueName = "myQueueName"
String routingKey = "#"
String exchange = "myExchange"
Queue queue = new Queue(queueName)
admin.declareQueue(queue)
TopicExchange exchange = new TopicExchange(exchange)
admin.declareExchange(exchange)
admin.declareBinding( BindingBuilder.bind(queue).to(exchange).with(routingKey) )
// normally passed to this method, moved to local var for simplicity
MyUpdater listener = new MyUpdater()
SimpleMessageListenerContainer container =
new SimpleMessageListenerContainer(rabbitMQConnectionFactory)
MessageListenerAdapter adapter = new MessageListenerAdapter(listener)
adapter.setMessageConverter(new SimpleMessageConverter())
container.setMessageListener(adapter)
container.setQueueNames(queueName)
container.start()
}