I have a multinode (2 nodes) setup of Weblogic 12c. In that I have deployed a spring boot application having microservices. The spring boot application is having the structure like Controller <--> Service <--> DAO. The #Cacheable annotation is used for few methods in "Service" classes/ layer. When endpoint it called, it reaches to controller and then it calls service class method. The issue is some of the calls are failing with below exception.
2023-01-15T14:55:31,122 ERROR [tuning)'] iServiceExceptionHandlerAdvice [, , ]
| Exception is encountered, message - WrongTarget! local: Member [XX.XX.XX.XX]:5981 - 9bXXX334-bf12-4XXb-afb6-XXXXXXX05442 this, expected-target: null, partitionId: 121, replicaIndex: 0, operation: com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
com.hazelcast.spi.exception.WrongTargetException: WrongTarget! local: Member [XX.XX.XX.XX]:5981 - 9bXXX334-bf12-4XXb-afb6-XXXXXXX05442 this, expected-target: null, partitionId: 121, replicaIndex: 0, operation: com.hazelcast.map.impl.operation.GetOperation, service: hz:impl:mapService
at com.hazelcast.spi.impl.operationservice.impl.Invocation.newTargetNullException(Invocation.java:336) ~[hazelcast-3.12.10.jar:3.12.10]
at com.hazelcast.spi.impl.operationservice.impl.PartitionInvocation.newTargetNullException(PartitionInvocation.java:94) ~[hazelcast-3.12.10.jar:3.12.10]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.initInvocationTarget(Invocation.java:289) ~[hazelcast-3.12.10.jar:3.12.10]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:614) ~[hazelcast-3.12.10.jar:3.12.10]
at com.hazelcast.spi.impl.operationservice.impl.Invocation.access$400(Invocation.java:96) ~[hazelcast-3.12.10.jar:3.12.10]
at com.hazelcast.spi.impl.operationservice.impl.Invocation$InvocationRetryTask.run(Invocation.java:827) ~[hazelcast-3.12.10.jar:3.12.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_141]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_141]
The above exception is from one node. Other node also fails intermittently with exact same exception, only the local: Member [XX.XX.XX.XX]:5982 will be different port like incremented by 1. This behavior goes away when I restart the deployed application (the spring boot application). It does not need to restart the complete weblogic server. Moreover this sometimes starts working without doing anything.
As per documentation of Hazelcast, https://docs.hazelcast.org/docs/3.7.8/manual/html-single/index.html says
WrongTargetException: Thrown when an operation is executed on the wrong machine, usually because the partition that operation belongs to has been moved to some other member.
There is no clear pattern to replicate this issue on local environment.
The alternatives I am thinking are
I can add try-catch in controller and log the details and throw some graceful exception. But still the data will be empty.
In try-catch in controller, I can call another method performing same operation but will not have #Cachable annotation. So that it will do everything same and fetch the data. put it in response and send it back.
Get rid of Hazelcast cache and use some other cache.
Is there any setting in Hazelcast which can do this (something similar to option 2) with some configuration. Instead of failing completely, it will do the invocation of method?
Related
I've been back and forth with an issue on our system that even with some research around the forums and several tests, we can't seem to be able to fix.
I'll try to be as clear as I can with what we are dealing with
We have a main service with a route that reads from an activemq queue ( spring boot with embedded broker ) sends it to a Route(B) and then ships everything to a final Route(C) . Route(B) is on a dependency of the service.
Camel Version: 3.3.0
Spring-boot version: 2.3.3.RELEASE
Route A:
onException(Exception::class.java)
.handled(true)
.bean("foo.ErrorProcessor", "processError")
from("activemq:queue:myqueue")
.routeId("myroute")
.to("direct:my_external_route")
.to(ExchangePattern.InOnly,"direct:myroute_result")
Route B:
onException(Exception::class.java)
.handled(true)
.bean("foo.ErrorProcessor", "processError")
from("direct:my_external_route")
.routeId("my_external_route")
.process {something()} //This processor can throw exceptions that are treated in our processor
Route C:
from("direct:myroute_result")
.process(someProcess())
.to(ExchangePattern.InOnly,"activemq:queue:results_queue")
Spring Boot activemq configs
spring:
jmx:
enabled: true
activemq:
broker-url: vm://localhost?broker.persistent=false,useShutdownHook=false
in-memory: true
non-blocking-redelivery: true
packages:
trust-all: false
trusted: com.mypackage
pool:
block-if-full: true
block-if-full-timeout: -1
enabled: false
idle-timeout: 30000
max-connections: 10
time-between-expiration-check: -1
use-anonymous-producers: true
Everything runs very well and smoothly when B's processors do not throw exceptions. When it does, even though they are being treated and a normal object is being returned in the message body, all we have on the logs is
2021-04-10 15:33:32.354 DEBUG [#1 - JmsConsumer[consumerName]] o.a.c.p.Pipeline
: Message exchange has failed: so breaking out of pipeline for exchange: Exchange[ID-1234] Handled by the error handler. {}
We even added a default error handler to our activemq connection factory but nothing happens there as well. We have a DLQ consumer who also does not seems to get anything. The error processor on routeA also does not catches anything which is expected since the exception was handled previously.
Has anyone ever had this issue or similar ? I know that some issues between Camel and the JMS component regarding error handling were raised in the past but we are struggling to understand what is the root of this issue.
Thanks in advance,
Pedro
Probably what you are looking for is the continued option on your Route B exception clause. This option allows you to continue routing to the original route as if the exception did not occur. Do not use the handled option as it will not allow routing to the original route but break out.
So your Route B should be defined as something like this:
onException(Exception::class.java) .continued(true)
.bean("foo.ErrorProcessor", "processError")
from("direct:my_external_route")
.routeId("my_external_route")
.process {something()}
Refer the camel documentation for more details: CAMEL EXCEPTION CLAUSE
At the company i work with we use Spring for Kafka without authentication and lately we did some experiments to setup the security in Kafka and we enabled authentication for a brief moment which cause a crush in all our consumers/producers within our microservices ! (the microservices stayed up)
The exception :
Authorization Exception and no authorizationExceptionRetryInterval set
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: foo-group
after some researchs we found out that this is the expected behavior by kafka clients and we needed to set the authorizationExceptionRetryInterval property
public void setAuthorizationExceptionRetryInterval​(java.time.Duration authorizationExceptionRetryInterval)
Set the interval between retries after AuthorizationException is
thrown by KafkaConsumer. By default the field is null and retries are
disabled. In such case the container will be stopped. The interval
must be less than max.poll.interval.ms consumer property.
Here is some other useful links
Setting authorizationExceptionRetryInterval for Spring Kafka
Why does the spring KafkaConsumer suspend all consumption from n topics when one fails to authorize
What i want to know is :
Is a failed authentication the only case when
consumers/producers goes down ?
If there are some other cases, how to make sure that our
consumers/producers recover without human intervention (restarting
the microservices) ? In other word how to check if the
consumers/producers are up and restart them otherwise ?
Containers are stopped only under the following circumstances:
AuthorizationException with no authorizationExceptionRetryInterval
NoOffsetForPartitionException - thrown when ConsumerConfig.AUTO_OFFSET_RESET_CONFIG is not earliest or latest and there is no existing offset for a partition with this consumer group.
FencedInstanceIdException - using transactions and static group members (meaning some other instance is using this instance id).
StopAfterFenceException - when stopContainerWhenFenced is true (default false) - only applies with transactions
Any Error (such as OOME)
I have Kubernetes running on two nodes and one application deployed on the two nodes (two pods, one per node).
It's a Spring Boot application. It uses OpenFeign for service discoverability. In the app i have a RestController defined and it has a few APIs and an #Autowired #Service which is called from inside the APIs.
Whenever i do a request on one of the APIs Kubernetes uses some sort of load-balancing to route the traffic to one of the pods, and the apps RestController is called. This is fine and i want this to be load-balanced.
The problem happens once that API is called and it calls the #Autowired #Service. Somehow this too gets load-balanced and the call to the #Service might end up on the other node.
Heres and example:
we have two nodes: node1, node2
we make a request to node1's IP address.
this might get load-balanced to node2 (this is fine)
node1 gets the request and calls the #Autowired #Service
the call jumps to node2 (this is where the problem happens)
And in code:
Controller:
#Autowired
private lateinit var userService: UserService
#PostMapping("/getUser")
fun uploadNewPC(#RequestParam("userId") userId: String): User {
println(System.getEnv("hostIP")) //123.45.67.01
return userService.getUser(userId)
}
Service:
#Service
class UserService {
fun getUser(userId: String) : User {
println(System.getEnv("hostIP")) //123.45.67.02
...
}
}
I want the load-balancing to happen only on the REST requests not the internal calls of the app to its #Service components. How would i achieve this? Is there any configuration to the way Spring Boot's #service components operate in Kubernetes clusters? Can i change this?
Thanks in advance.
Edit:
After some debugging i found that It wasn't the Service that was load balanced to another node but the initial http request. Even though the request is specifically send to the url of node1... And since i was debugging both nodes at the same time, i didn't notice this.
Well, I haven't used openfeign, but in my understanding it can loadbalance only REST requests indeed.
If I've got your question right, you say that when the REST controller calls the service component (UserService in this case) the network call is issued and this is undesirable.
In this case, I believe, following points for consideration will be beneficial:
Spring boot has nothing to do with load balancing at this level by default, it should be a configured in the spring boot application somehow.
This also has nothing to do with the fact that this application runs in a Kubernetes environment, again its only a spring boot configuration.
Assuming, you have a UserService interface that obviously doesn't have any load balancing logic, spring boot must wrap it into some kind of proxy that adds these capabilities. So try to debug the application startup, place a breakpoint in the controller method and check out what is the actual type of the user service, again it must be some sort of proxy
If the assumption in 3 is correct, there must be some kind of bean post processor (possibly in spring.factories file of some dependency) class that gets registered within the application context. Probably if you'll create some custom method that will print all beans (Bean Post Processor is also a bean), you'll see the suspicious bean.
I have a WAR application as follows:
JPA/Hibernate 4.1.9.Final
Hibernate Envers 4.1.9.Final
Spring 3.1.3.RELEASE
Spring MVC with JSON/REST
My frontend (web page) makes a request, this leads to a new entity to be saved (this seems to succeed in any case) and then Envers will save corresponding revision info.
Typical deployment is in Tomcat 7, where this works just fine.
For unit testing I spin up a Jetty (8.1.9.v20130131) programmatically, which fails. It baffles me why this behavior is different. I can only imagine that Jetty has a different (lesser standard, perhaps) transaction management on board than Tomcat, but I've failed to pin it down or otherwise work around this.
Below is how I create the web application programmatically:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration({"classpath:applicationContext.xml" })
public class AddCustomerTest {
And:
server = new Server(serverPort);
WebAppContext webAppContext = new WebAppContext();
webAppContext.setContextPath("/webapp");
webAppContext.setWar("src/main/webapp");
webAppContext.setServer(server);
server.setHandler(webAppContext);
Below is the essential exception that I'm getting:
Caused by: org.springframework.transaction.IllegalTransactionStateException: No existing transaction found for transaction marked with propagation 'mandatory'
at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:357)
at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:334)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy115.getByUuid(Unknown Source)
at com.totaalsoftware.incidentmanager.entity.audit.RevisionEntityListener.setUser(RevisionEntityListener.java:53)
at com.totaalsoftware.incidentmanager.entity.audit.RevisionEntityListener.instanceNewRevision(RevisionEntityListener.java:40)
at com.totaalsoftware.incidentmanager.entity.audit.RevisionEntityListener.newRevision(RevisionEntityListener.java:34)
at org.hibernate.envers.revisioninfo.DefaultRevisionInfoGenerator.generate(DefaultRevisionInfoGenerator.java:95)
at org.hibernate.envers.synchronization.AuditProcess.getCurrentRevisionData(AuditProcess.java:124)
at org.hibernate.envers.synchronization.AuditProcess.executeInSession(AuditProcess.java:106)
at org.hibernate.envers.synchronization.AuditProcess.doBeforeTransactionCompletion(AuditProcess.java:155)
at org.hibernate.engine.spi.ActionQueue$BeforeTransactionCompletionProcessQueue.beforeTransactionCompletion(ActionQueue.java:662)
... 80 more
My RevisionEntityListener looks up some user data (from the database, using Hibernate of course). Clearly there's no transaction available, but only when running in Jetty. I've tried marking the RevisionEntityListener transactional in various ways, to no avail.
Let me know if you need any other info. Your help much appreciated!
Stupid me...
The following two lines were copied over from some other test class:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration({"classpath:applicationContext.xml" })
These two lines served no purpose since the entire application was loaded through the instantiation and configuration of Jetty. But somehow the above clashed with the application in the Jetty server, probably due to being in the same JVM. Removing the above two lines from the test class fixed it!
I am getting the following Spring Framework error message:
Invocation of getLogoForGlobalConext() in class $Proxy44 threw exception
org.springframework.transaction.UnexpectedRollbackException:
Transaction rolled back because it has been marked as rollback-only
at template/includes/macros.vm line 1651, column 43
I opened macros.vm and looked up line 1651 and it looks like this:
#set ($globalLogo = $spaceManager.getLogoForGlobalContext());
From my research it looks like $Proxy44 is actually the $spaceManager variable (or an instance of DefaultSpaceManager.java).
This message appears randomly and when the web app tries to download an image/attachment sitting somewhere on the web server / database.
The attachmentmanager is governed by Spring's Transaction Management and the following transaction attributes are used when an image/attachment is downloaded:
propagation - for all methods in the attachment manager
propagation and read-only - for all methods in attachment manager starting with "get".
The attributes are defined in Spring Framework - Chapter 9. Transaction management.
What I am thinking is I need to set a timeout on the transactions (like set it to infinity).
It turns out one of the getter methods was performing a write to database. Specifically, it was updating the cache with some information every few minutes. When this update occurred, the UnexpectedRollbackException was thrown. Since this transaction is supposed to be "read only" as defined by the transaction attributes mentioned above, we are not allowed to perform updates during a getter operation.
I changed the getter method to not perform any updates to cache and to simply use the cache even if it expired, and the error goes away.
Hope this helps someone else.