I have the following scenario:
- a Hazelcast Server as a microservice which performs some computations when receives a method call.
- a Hazelcast Client as another microservice which calls the Hazelcast Server through the specified method call.
I want that when I throw an exception from the Hazelcast Server to receive it on the Hazelcast Client side as it is (currently, I'm receiving somthing like this: java.util.concurrent.ExecutionException: com.hazelcast.client.UndefinedErrorCodeException: Class name: ro.orange.eshop.personalisationengineapi.application.exception.ValidationException)
I've digged a little into the APIs and on the Hazelcast Client side I've found a way to register a new exception:
#Bean
fun addHazelcastKnownExceptions(hazelcastInstance: HazelcastInstance): Int {
val hazelcastClientInstance = (hazelcastInstance as HazelcastClientProxy).client
hazelcastClientInstance.clientExceptionFactory.register(400, ValidationException::class.java) { message, cause -> ValidationException(message, cause) }
return 1
}
But it seems that this exception must be registered also on the server side as well. And here comes the problem! On the server side, I've found a class called ClientExceptions which has a method public void register(int errorCode, Class clazz) but I can't find a way to receive a ClientExceptions instance (I should mention that I'm using Hazelcast Spring).
Thank you!
It is not supported to register custom exception factory as an API as of 3.12.x.
Related issue to follow https://github.com/hazelcast/hazelcast/issues/9753
As a workaround, I could suggest using class name (UndefinedErrorCodeException.getOriginialClassName()) to recreate exception classes on the client side.
== EDIT ==
Client API does not support it. You have found the private API.
If you are ok with relying on private API here is the hack for registering classes on the hazelcast server:
Note that I DO NOT recommend this solution since it relies on private API that can change.
HazelcastInstance instance = Hazelcast.newHazelcastInstance();
if (instance instanceof HazelcastInstanceProxy) {
HazelcastInstanceImpl original = ((HazelcastInstanceProxy) instance).getOriginal();
ClientExceptions clientExceptions = original.node.getClientEngine().getClientExceptions();
clientExceptions.register( USER_EXCEPTIONS_RANGE_START + 1, UndefinedCustomFormatException.class);
}
Related
We have Spring Boot services running in Kubernetes and are using the Spring Cloud Kubernetes Load Balancer functionality with RestTemplate to make calls to other Spring Boot services. One of the main reasons we have this in place is historical - in that previously we ran our services in EC2 using Eureka for service discovery and after the migration we kept the Spring discovery client/client-side load balancing in place (updating dependencies etc for it to work with the Spring Cloud Kubernetes project)
We have a problem that when one of the target pods goes down we get multiple failures for requests for a period of time with java.net.NoRouteToHostException ie the spring load balancer is still trying to send to that pod.
So I have a few questions on this:
Shouldn't the target instance get removed automatically when this happens? So it might happen once but after that, the target pod list will be repaired?
Or if not is there some other configuration we need to add to handle this - eg retry / circuit breaker, etc?
A more general question is what benefit does Spring's client-side load balancing bring with Kubernetes? Without it, our service would still be able to call other services using Kubernetes built-in service / load-balancing functionality and this should handle the issue of pods going down automatically. The Spring documentation also talks about being able to switch from POD mode to SERVICE mode (https://docs.spring.io/spring-cloud-kubernetes/docs/current/reference/html/index.html#loadbalancer-for-kubernetes). But isn't this service mode just what Kubernetes does automatically? I'm wondering if the simplest solution here isn't to remove the Spring Load Balancer altogether? What would we lose then?
An update on this: we had the spring-retry dependency in place, but the retry was not working as by default it only works for GETs and most of our calls are POST (but OK to call again). Adding the configuration spring.cloud.loadbalancer.retry.retryOnAllOperations: true fixed this, and hence most of these failures should be avoided by the retry using an alternative instance on the second attempt.
We have also added a RetryListener that clears the load balancer cache for the service on certain connection exceptions:
#Configuration
public class RetryConfig {
private static final Logger logger = LoggerFactory.getLogger(RetryConfig.class);
// Need to use bean factory here as can't autowire LoadBalancerCacheManager -
// - it's set to 'autowireCandidate = false' in LoadBalancerCacheAutoConfiguration
#Autowired
private BeanFactory beanFactory;
#Bean
public CacheClearingLoadBalancedRetryFactory cacheClearingLoadBalancedRetryFactory(ReactiveLoadBalancer.Factory<ServiceInstance> loadBalancerFactory) {
return new CacheClearingLoadBalancedRetryFactory(loadBalancerFactory);
}
// Extension of the default bean that defines a retry listener
public class CacheClearingLoadBalancedRetryFactory extends BlockingLoadBalancedRetryFactory {
public CacheClearingLoadBalancedRetryFactory(ReactiveLoadBalancer.Factory<ServiceInstance> loadBalancerFactory) {
super(loadBalancerFactory);
}
#Override
public RetryListener[] createRetryListeners(String service) {
RetryListener cacheClearingRetryListener = new RetryListener() {
#Override
public <T, E extends Throwable> boolean open(RetryContext context, RetryCallback<T, E> callback) { return true; }
#Override
public <T, E extends Throwable> void close(RetryContext context, RetryCallback<T, E> callback, Throwable throwable) {}
#Override
public <T, E extends Throwable> void onError(RetryContext context, RetryCallback<T, E> callback, Throwable throwable) {
logger.warn("Retry for service {} picked up exception: context {}, throwable class {}", service, context, throwable.getClass());
if (throwable instanceof ConnectTimeoutException || throwable instanceof NoRouteToHostException) {
try {
LoadBalancerCacheManager loadBalancerCacheManager = beanFactory.getBean(LoadBalancerCacheManager.class);
Cache loadBalancerCache = loadBalancerCacheManager.getCache(CachingServiceInstanceListSupplier.SERVICE_INSTANCE_CACHE_NAME);
if (loadBalancerCache != null) {
boolean result = loadBalancerCache.evictIfPresent(service);
logger.warn("Load Balancer Cache evictIfPresent result for service {} is {}", service, result);
}
} catch(Exception e) {
logger.error("Failed to clear load balancer cache", e);
}
}
}
};
return new RetryListener[] { cacheClearingRetryListener };
}
}
}
Are there any issues with this approach? Could something like this be added to the built in functionality?
Shouldn't the target instance get removed automatically when this
happens? So it might happen once but after that the target pod list
will be repaired?
To resolve this issue you have to use the Readiness and Liveness Probe in Kubernetes.
Readiness will check the health of the endpoint that your application has, on the period of interval. If the application fails it will mark your PODs as Unready to accept the Traffic. So no traffic will go to that POD(replica).
Liveness will restart your application if it fails so your container or we can say POD will come up again and once we will get 200 response from app K8s will mark your POD as Ready to accept the traffic.
You can create the simple endpoint in the application that give response as 200 or 204 as per need.
Read more at : https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Make sure you application using the Kubernetes service to talk with each other.
Application 1 > Kubernetes service of App 2 > Application 2 PODs
To enable load balancing based on Kubernetes Service name use the
following property. Then load balancer would try to call application
using address, for example service-a.default.svc.cluster.local
spring.cloud.kubernetes.loadbalancer.mode=SERVICE
The most typical way to use Spring Cloud LoadBalancer on Kubernetes is
with service discovery. If you have any DiscoveryClient on your
classpath, the default Spring Cloud LoadBalancer configuration uses it
to check for service instances. As a result, it only chooses from
instances that are up and running. All that is needed is to annotate
your Spring Boot application with #EnableDiscoveryClientto enable
K8s-native Service Discovery.
References : https://stackoverflow.com/a/68536834/5525824
I am using activiti 5.18.
Behind the scenes : There are few task which are getting routed though a workflow. Some of these tasks are eligible for escalation. I have written my escalation listener as follows.
#Component
public class EscalationTimerListener implements ExecutionListener {
#Autowired
ExceptionWorkflowService exceptionWorkflowService;
#Override
public void notify(DelegateExecution execution) throws Exception {
//Process the escalated tasks here
this.exceptionWorkflowService.escalateWorkflowTask(execution);
}
}
Now when I start my tomcat server activiti framework internally calls the listener even before my entire spring context is loaded. Hence exceptionWorkflowService is null (since spring hasn't inejcted it yet!) and my code breaks.
Note : this scenario only occurs if my server isn't running at the escalation time of tasks and I start/restart my server post this time. If my server is already running during escalation time then the process runs smoothly. Because when server started it had injected the service and my listener has triggered later.
I have tried delaying activiti configuration using #DependsOn annotation so that it loads after ExceptionWorkflowService is initialized as below.
#Bean
#DependsOn({ "dataSource", "transactionManager","exceptionWorkflowService" })
public SpringProcessEngineConfiguration getConfiguration() {
final SpringProcessEngineConfiguration config = new SpringProcessEngineConfiguration();
config.setAsyncExecutorActivate(true);
config.setJobExecutorActivate(true);
config.setDataSource(this.dataSource);
config.setTransactionManager(this.transactionManager);
config.setDatabaseSchemaUpdate(this.schemaUpdate);
config.setHistory(this.history);
config.setTransactionsExternallyManaged(this.transactionsExternallyManaged);
config.setDatabaseType(this.dbType);
// Async Job Executor
final DefaultAsyncJobExecutor asyncExecutor = new DefaultAsyncJobExecutor();
asyncExecutor.setCorePoolSize(2);
asyncExecutor.setMaxPoolSize(50);
asyncExecutor.setQueueSize(100);
config.setAsyncExecutor(asyncExecutor);
return config;
}
But this gives circular reference error.
I have also tried adding a bean to SpringProcessEngineConfiguration as below.
Map<Object, Object> beanObjectMap = new HashMap<>();
beanObjectMap.put("exceptionWorkflowService", new ExceptionWorkflowServiceImpl());
config.setBeans(beanObjectMap);
and the access the same in my listener as :
Map<Object, Object> registeredBeans = Context.getProcessEngineConfiguration().getBeans();
ExceptionWorkflowService exceptionWorkflowService = (ExceptionWorkflowService) registeredBeans.get("exceptionWorkflowService");
exceptionWorkflowService.escalateWorkflowTask(execution);
This works but my repository has been autowired into my service which hasn't been initialized yet! So it again throws error in service layer :)
So is there a way that I can trigger escalation listeners only after my entire spring context is loaded?
Have you tried binding the class to ApplicationListener?
Not sure if it will work, but equally I'm not sure why your listener code is actually being executed on startup.
Try to set the implementation type of listeners using Java class or delegate expression and then in the class implement JavaDelegate instead of ExecutionListener.
I have method in a Spring component which receives messages from a Spring Integration channel. When a message is received, it is sent to a WebSocket endpoint. This doesn't work. The message is not broadcast.
this.messagingTemplate.convertAndSend("/topic/update", dto);
However when I put the same code inside a Web Controller and put a RequestMapping on it, and call that endpoint, it works. The message is broadcast.
What might be causing it not to work, when it is called by the Spring integration executor?
when it works: .14:01:19.939 [http-nio-8080-exec-4] DEBUG o.s.m.s.b.SimpleBrokerMessageHandler - Processing MESSAGE destination=/topic/update session=null payload={XXX}
.14:01:19.939 [http-nio-8080-exec-4] DEBUG o.s.m.s.b.SimpleBrokerMessageHandler - Broadcasting to 1 sessions.
when it doesnt work, second message is not there. (thread is taskExecutor-1 instead of http-nio..)
Controller code:
#RequestMapping("/testreq")
public void updateDelta() {
SummaryDTO dto = new SummaryDTO();
dto.setValue(-5000.0);
dto.setName("G");
this.messagingTemplate.convertAndSend("/topic/update", dto);
}
//this method is called by Spring Integration
//created by serviceActivator = new
//ServiceActivatingHandler(webcontroller,"update");
public void updateDelta(SummaryDTO dto) {
this.messagingTemplate.convertAndSend("/topic/update", dto);
}
message send:
synchronized(this){
...
this.updatedcontrollerchannel.send(MessageBuilder.withPayload(summarydto).build(
));
}
channel creation:
updatedchannel = new DirectChannel();
updatedchannel.setBeanName("updatedcontroller");
serviceActivator = new ServiceActivatingHandler(detailService,"update");
handlerlist.add(serviceActivator);
updatedchannel.subscribe(serviceActivator);
beanFactory.registerSingleton("updatedcontroller", channel);
UPDATE
I added spring messaging source code to my environment and realized the following: There are 2 instances of the SimpleBrokerMessageHandler class in the runtime. For the working copy subscriptionregistry has one entry and for the nonworking one, it has 0 subscriptions. Does this give a clue for the root cause of the problem? There is only one MessageSendingOperations variable defined and it is on the controller.
i found the cause of the problem. Class which has #EnableWebSocketMessageBroker annotation was loaded twice and it caused two instances of SimpleBrokerMessageHandler to be created. #Artem Bilan: thanks for your time.
Should be the problem with the non-properly injected SimpMessageSendingOperations.
This one is populated by the AbstractMessageBrokerConfiguration.brokerMessagingTemplate() #Bean.
However I would like to suggest you to take a look into the WebSocketOutboundMessageHandler from Spring Integration: https://docs.spring.io/spring-integration/docs/4.3.12.RELEASE/reference/html/web-sockets.html
UPDATE
This works for me in the test-case:
#Bean
#InboundChannelAdapter(channel = "nullChannel", poller = #Poller(fixedDelay = "1000"))
public Supplier<?> webSocketPublisher(SimpMessagingTemplate brokerMessagingTemplate) {
return () -> {
brokerMessagingTemplate.convertAndSend("/topic/foo", "foo");
return "foo";
};
}
And I have this DEBUG logs:
12:57:27.606 DEBUG [task-scheduler-1][org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler] Processing MESSAGE destination=/topic/foo session=null payload=foo
12:57:27.897 DEBUG [clientInboundChannel-2][org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler] Processing SUBSCRIBE /topic/foo id=subs1 session=941a940bf07c47a1ac786c1adfdb6299
12:57:40.797 DEBUG [task-scheduler-1][org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler] Processing MESSAGE destination=/topic/foo session=null payload=foo
12:57:40.798 DEBUG [task-scheduler-1][org.springframework.messaging.simp.broker.SimpleBrokerMessageHandler] Broadcasting to 1 sessions.
Everything works well from Spring Integration.
That's why I asked your whole Spring Boot app to play from our side.
UPDATE 2
When you develop Web application be sure to merge all the configs contexts to a single one application context - WebApplicationContext:
If an application context hierarchy is not required, applications may return all configuration via getRootConfigClasses() and null from getServletConfigClasses().
See more info in the Spring Framework Reference Manual.
maybe someone has an idea to my following problem:
I am currently on a project, where i want to use the AWS SQS with Spring Cloud integration. For the receiver part i want to provide a API, where a user can register a "message handler" on a queue, which is an interface and will contain the user's business logic, e.g.
MyAwsSqsReceiver receiver = new MyAwsSqsReceiver();
receiver.register("a-queue-name", new MessageHandler(){
#Override
public void handle(String message){
//... business logic for the received message
}
});
I found examples, e.g.
https://codemason.me/2016/03/12/amazon-aws-sqs-with-spring-cloud/
and read the docu
http://cloud.spring.io/spring-cloud-aws/spring-cloud-aws.html#_sqs_support
But the only thing i found there to "connect" a functionality for processing a incoming message is a annotation on a method, e.g. #SqsListener or #MessageMapping.
These annotations are fixed to a certain queue-name, though. So now i am at a loss, how to dynamically "connect" my provided "MessageHandler" (from my API) to the incoming message for the specified queuename.
In the Config the example there is a SimpleMessageListenerContainer, which gets a QueueMessageHandler set, but this QueueMessageHandler does not seem
to be the right place to set my handler or to override its methods and provide my own subclass of QueueMessageHandler.
I already did something like this with the Spring Amqp integration and RabbitMq and thought, that it would be also similar here with AWS SQS.
Does anyone have an idea, how to accomplish this?
thx + bye,
Ximon
EDIT:
I found, that Spring JMS could actually do that, e.g. www.javacodegeeks.com/2016/02/aws-sqs-spring-jms-integration.html. Does anybody know, what consequences using JMS protocol has here, good or bad?
I am facing the same issue.
I am trying to go in an unusual way where I set up an Aws client bean at build time and then instead of using sqslistener annotation to consume from the specific queue I use the scheduled annotation which I can programmatically pool (each 10 secs in my case) from which queue I want to consume.
I did the example that iterates over queues defined in properties and then consumes from each one.
Client Bean:
#Bean
#Primary
public AmazonSQSAsync awsSqsClient() {
return AmazonSQSAsyncClientBuilder
.standard()
.withRegion(Regions.EU_WEST_1.getName())
.build();
}
Consumer:
// injected in the constructor
private final AmazonSQSAsync awsSqsClient;
#Scheduled(fixedDelay = 10000)
public void pool() {
properties.getSqsQueues()
.forEach(queue -> {
val receiveMessageRequest = new ReceiveMessageRequest(queue)
.withWaitTimeSeconds(10)
.withMaxNumberOfMessages(10);
// reading the messages
val result = awsSqsClient.receiveMessage(receiveMessageRequest);
val sqsMessages = result.getMessages();
log.info("Received Message on queue {}: message = {}", queue, sqsMessages.toString());
// deleting the messages
sqsMessages.forEach(message -> {
val deleteMessageRequest = new DeleteMessageRequest(queue, message.getReceiptHandle());
awsSqsClient.deleteMessage(deleteMessageRequest);
});
});
}
Just to clarify, in my case, I need multiple queues, one for each tenant, with the queue URL for each one passed in a property file. Of course, in your case, you could get the queue names from another source, maybe a ThreadLocal which has the queues you have created in runtime.
If you wish, you can also try the JMS approach where you create message consumers and add a listener to each one you wish (See the doc Aws Jms documentation).
When we do Spring and SQS we use the spring-cloud-starter-aws-messaging.
Then just create a Listener class
#Component
public class MyListener {
#SQSListener(value="myqueue")
public void listen(MyMessageType message) {
//process the message
}
}
I have JAX-WS container-less service (published via Endpoint.publish() right from main() method). I want my service to validate input messages. I have tried following annotation: #SchemaValidation(handler=MyErrorHandler.class) and implemented an appropriate class. When I start the service, I get the following:
Exception in thread "main" javax.xml.ws.WebServiceException:
Annotation #com.sun.xml.internal.ws.developer.SchemaValidation(outbound=true,
inbound=true, handler=class mypackage.MyErrorHandler) is not recognizable,
atleast one constructor of class
com.sun.xml.internal.ws.developer.SchemaValidationFeature
should be marked with #FeatureConstructor
I have found few solutions on the internet, all of them imply the use of WebLogic container. I can't use container in my case, I need embedded service. Can I still use schema validation?
The #SchemaValidation annotation is not defined in the JAX-WS spec, but validation is left open. This means you need something more than only the classes in the jdk.
As long as you are able to add some jars to your classpath, you can set this up pretty easily using metro (which is also included in WebLogic. This is why you find solutions that use WebLogic as container.). To be more precise, you need to add two jars to your classpath. I'd suggest to
download the most recent metro release.
Unzip it somewhere.
Add the jaxb-api.jar and jaxws-api.jar to your classpath. You can do this for example by putting them into the JAVA_HOME/lib/endorsed or by manually adding them to your project. This largely depends on the IDE or whatever you are using.
Once you have done this, your MyErrorHandler should work even if it is deployed via Endpoint.publish(). At least I have this setup locally and it compiles and works.
If you are not able to modify your classpath and need validation, you will have to validate the request manually using JAXB.
Old question, but I solved the problem using the correct package and minimal configuration, as well using only provided services from WebLogic. I was hitting the same problem as you.
Just make sure you use correct java type as I described here.
As I am planning to expand to a tracking mechanism I also implemented the custom error handler.
Web Service with custom validation handler
import com.sun.xml.ws.developer.SchemaValidation;
#Stateless
#WebService(portName="ValidatedService")
#SchemaValidation(handler=MyValidator.class)
public class ValidatedService {
public ValidatedResponse operation(#WebParam(name = "ValidatedRequest") ValidatedRequest request) {
/* do business logic */
return response;
}
}
Custom Handler to log and store error in database
public class MyValidator extends ValidationErrorHandler{
private static java.util.logging.Logger log = LoggingHelper.getServerLogger();
#Override
public void warning(SAXParseException exception) throws SAXException {
handleException(exception);
}
#Override
public void error(SAXParseException exception) throws SAXException {
handleException(exception);
}
#Override
public void fatalError(SAXParseException exception) throws SAXException {
handleException(exception);
}
private void handleException(SAXParseException e) throws SAXException {
log.log(Level.SEVERE, "Validation error", e);
// Record in database for tracking etc
throw e;
}
}