Hazelcast Cache Manager: Cannot overwrite a Cache's CacheManager - spring

On an application I am working I am trying to upgrade from Hazelcast 3.6 to 3.12.4 and I am encountering some problems which reproduce easily when two or more tests are ran together. The tests are all annotated with #WebAppConfiguration and include the Spring's application configuration using ContextConfiguration(classes = {AppConfig.class})
As part of the configuration, I have a #Bean that called CacheAwareStorage that initiates the CacheManager. THe initialization is quite basic:
public Cache<T, V> initCache(String name, Class<T> type, Class<T> valueType) {
Cache<T, V> cache = manager.getCache(cacheName, keyType, valueType);
if (cache != null)
{
return cache;
}
cache = manager.createCache(cacheName, config);
return cache;
}
The problem occurs when the context is refreshed as part of the test suit, which I think is done in AbstractTestNGSpringContextTests since I don't explicitly refresh the context. The following error occurs which result sin only the first class of tests to pass:
GenericWebApplicationContext: Refreshing org.springframework.web.context.support.GenericWebApplicationContext#6170989a
....
WARN GenericWebApplicationContext: Exception encountered during context initialization - cancelling refresh attempt
....
Factory method 'tokenStore' threw exception
nested exception is java.lang.IllegalStateException: Cannot overwrite a Cache's CacheManager.
Looking over what has changed, I see that the AbstracthazelcastCacheManager throws an IllegalStateException which comes from the Hazelcast CacheProxy. To be more precise, the manager.getCache() -> getCacheUnchecked() -> creates a cache proxy in createCacheProxy() -> and set's the proxy's manager to the current manager in cacheProxy.setCacheManager().
Starting with Hazelcast v3.9, this is no longer allowed once the manager has already been set.
What would be a solution for this? It may be that there is a bug in Hazelcast (there is no check if the manager that is being set is actually different than the already existing one), however I am looking for something that I can do on my side. Why the 'getCache()' tries to re-create the proxy is another thing that I do not understand.
I assume that I must do something so that the Context is not refreshed, however I don't know how (if at all) I can do that.

The problem was due to the way the Cache manager Bean was created. I used the internal Hazelcast cache manager and a new instance was created each time. Using the JCache API as bellow, solved the problem
#Bean
public CacheManager cacheManager() {
HazelcastServerCachingProvider provider = Caching.getCachingProvider(); // or add class name of Hazelcast server caching provider to disambiguate
return provider.getCacheManager(null, null, HazelcastCachingProvider.propertiesByInstanceItself(HAZELCAST_INSTANCE));
}
Help received from Hazelcast team on this: https://github.com/hazelcast/hazelcast/issues/16212

Related

Redis cache metrics with Prometheus(Spring boot)

I am using RedisTemplate for caching purpose in my spring boot service. Now I want to check cache hit/cache miss through end point actuator/prometheus. But can not see cache hit/cache miss for the cache.
The code I have written is something like below
#EnableCaching
#Configuration
public class CachingConfiguration {
#Bean
public RedisTemplate<String, SomeData> redisTemplate(LettuceConnectionFactory connectionFactory, ObjectMapper objectMapper)
{
RedisTemplate<String, SomeData> template = new RedisTemplate<>();
template.setConnectionFactory(connectionFactory);
var valueSerializer = new Jackson2JsonRedisSerializer<SomeData>(SomeData.class);
valueSerializer.setObjectMapper(objectMapper);
template.setValueSerializer(valueSerializer);
return template;
}
}
Now am doing like below to get and save into cache
to get:-
redisTemplate.opsForValue().get(key);
And to save:-
redisTemplate.opsForValue().set(key, obj, some_time_limit);
My cache is working properly, am getting able to save into cache and getting proper data.
But I don't see cache hit/miss related data inside actuator/prometheus.
In my application.yml file I have added below
cache:
redis:
enable-statistics: 'true'
I would assume that in order for Springboot Cache Monitoring to apply (Including Hits/Misses), you would need to depend on AutoConfiguration.
In your case you are creating the RedisTemplate yourself, and probably enable-statistics is not actually applied.
Can you remove the redistemplate creation and use #Cacheable annotation abstraction? That way any supported Cache library will work out of the box, without you having to create #Bean and manually configuring it.
Otherwise, generally if you wanted to enable statistics on a cache manager manually, you will need to call RedisCacheManager.RedisCacheManagerBuilder enableStatistics():
https://docs.spring.io/spring-data/redis/docs/current/api/org/springframework/data/redis/cache/RedisCacheManager.RedisCacheManagerBuilder.html
For Reference:
Auto-configuration enables the instrumentation of all available Cache
instances on startup, with metrics prefixed with cache. Cache
instrumentation is standardized for a basic set of metrics.
Additional, cache-specific metrics are also available.
Metrics are tagged by the name of the cache and by the name of the
CacheManager, which is derived from the bean name.
Only caches that are configured on startup are bound to the registry. For caches not
defined in the cache’s configuration, such as caches created on the
fly or programmatically after the startup phase, an explicit
registration is required. A CacheMetricsRegistrar bean is made
available to make that process easier.
I had exactly the same question and spent a good number of hours trying to figure out how to enable cache metrics for my manually created RedisTemplate instance.
What I eventually realised is that it's only RedisCache class which collects and exposes CacheStatistics through getStatistics() method. As far as I can see there is nothing like that for RedisTemplate, which means you either need to switch to using RedisCache through RedisCacheManager and #Cacheable annotation or implement your custom metrics collection.

Deploying Spring Integration to WebSphere ND 8.5.5

I am looking for some guidance on deploying a simple Spring Integration application to WebSphere. The overall scope of the application is quite simple - it reads from a RabbitMQ endpoint, transforms any messages received to a specific xml format, and then posts the message to a JMS endpoint in WAS.
Initially, I built the application as a JAR. I was able to get it to work well enough with SSL turned off on the IIOP endpoints in WAS, but despite hours of debugging I never could get it to communicate properly with WAS over SSL. The initial handshake and communication with the bootstrap port was successful, but the SIB endpoint rejected the exact same certificate chain with the usual PKIX chaining error, and no amount of certificate importing made any difference.
So I elected to work out deploying the application as a web app into WAS itself, which would have been the end goal anyways. This caused a number of issues that i've had to work through:
I have not gotten properties to work in the normal Spring fashion. I assume that in this context Spring needs to be explicitly told where to look, but i've sidestepped this with hardcoding for now. Perhaps using #Resource annotations would be the way to do this in this context?
Various jar versioning issues, which i've mostly worked out by setting the application classloader as PARENT_LAST, and judiciously removing things that seemed redundant.
Oddly I did have to add some jars related to Parameter validation which don't seem to have been present in my original maven build.
Needing to set some values in the web.xml in order for spring to location configuration beans, specifically setting init-param with contextClass (org.springframework.web.context.support.AnnotationConfigWebApplicationContext) and contextConfigLocation set to a list of the objects that would normally be loaded via the #Configuration annotation.
May or may not be necessary but I did move from Maven to IID in order to hopefully avoid versioning issues with IBM related jars.
Now I would like to know if there are other items generally needed to be done to deploy Spring (especially Spring Integration) to WAS, and whether the above seems like enough.
In addition, I have an issue with the actual JMS connection to WAS. I have tried to use the UserCredentialsConnectionFactoryAdapter, and was successful with this with Spring standalone. However when deployed in WAS, an exception is thrown:
Caused by: java.lang.ClassCastException: com.ibm.ws.sib.api.jms.impl.JmsManagedQueueConnectionFactoryImpl incompatible with javax.jms.ConnectionFactory
I believe this is thrown when the setTargetConnectionFactory method is called, since if I use the connection factory without the UserCredentialsConnectionFactoryAdapter, it works fine, except the connection by "anonymous" is rejected by the bus:
[03/03/21 15:23:32:934 EST] 0000016c SibMessage W [BPM.WorkflowServer.Bus:Node1.server1-BPM.WorkflowServer.Bus] CWSII0212W: The bus BPM.WorkflowServer.Bus denied an anonymous user access to the bus.
If you want to see the code, this works fine (but doesn't authenticate):
#Bean
public ConnectionFactory jmsConnectionFactory() throws NamingException {
ConnectionFactory connectionFactory = null;
Context ctx = null;
Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY, "com.ibm.websphere.naming.WsnInitialContextFactory");
p.put(Context.PROVIDER_URL, providerUrl);
p.put(Context.SECURITY_AUTHENTICATION,"simple");
p.put(Context.SECURITY_PRINCIPAL,jmsUsername);
p.put(Context.SECURITY_CREDENTIALS,jmsPassword);
ctx = new InitialContext(p);
if (null != ctx)
System.out.println("Got naming context");
connectionFactory = (QueueConnectionFactory) ctx.lookup("javax.jms.QueueConnectionFactory");
if (null != connectionFactory)
System.out.println("Got connection factory");
return connectionFactory;
}
Whereas this throws the class cast exception:
#Bean
public UserCredentialsConnectionFactoryAdapter jmsConnectionFactory() throws NamingException {
ConnectionFactory connectionFactory = null;
Context ctx = null;
Properties p = new Properties();
p.put(Context.INITIAL_CONTEXT_FACTORY, "com.ibm.websphere.naming.WsnInitialContextFactory");
p.put(Context.PROVIDER_URL, providerUrl);
p.put(Context.SECURITY_AUTHENTICATION,"simple");
p.put(Context.SECURITY_PRINCIPAL,jmsUsername);
p.put(Context.SECURITY_CREDENTIALS,jmsPassword);
ctx = new InitialContext(p);
if (null != ctx)
System.out.println("Got naming context");
connectionFactory = (ConnectionFactory) ctx.lookup("javax.jms.QueueConnectionFactory");
if (null != connectionFactory)
System.out.println("Got connection factory");
UserCredentialsConnectionFactoryAdapter adapter = new UserCredentialsConnectionFactoryAdapter();
adapter.setTargetConnectionFactory(connectionFactory);
adapter.setUsername(jmsUsername);
adapter.setPassword(jmsPassword);
return adapter;
// return connectionFactory;
}
Note: the credentials set in the Context properties seem to have no effect.
I am using this connection factory with Spring Integration Java DSL:
.handle(Jms.outboundAdapter(jmsConfig.jmsConnectionFactory())
.destination(jmsDestination))
I understand from WebSphere documentation that supplying credentials happens on the ConnectionFactory.getConnection() call. So I wonder whether there is any hook in the DSL where I could override the getConnection so as to provide parameters and avoid the class cast exception that I am seeing.
Alternately I am considering just explicitly calling jms template methods to send the message using a lambda in the handler and creating the connection manually.
So, finally what I would like to ask for is:
Any overall guidance on deploying a Spring application to WebSphere traditional
What may be causing the class cast exception
ps, I have placed all of the spring, et al jars in a shared library. This is the contents:
c:/IBM/IID/sharedlibs/spring/accessors-smart-1.2.jar
c:/IBM/IID/sharedlibs/spring/amqp-client-5.10.0.jar
c:/IBM/IID/sharedlibs/spring/android-json-0.0.20131108.vaadin1.jar
c:/IBM/IID/sharedlibs/spring/apiguardian-api-1.1.0.jar
c:/IBM/IID/sharedlibs/spring/asm-5.0.4.jar
c:/IBM/IID/sharedlibs/spring/assertj-core-3.18.1.jar
c:/IBM/IID/sharedlibs/spring/byte-buddy-1.10.19.jar
c:/IBM/IID/sharedlibs/spring/byte-buddy-agent-1.10.19.jar
c:/IBM/IID/sharedlibs/spring/hamcrest-2.2.jar
c:/IBM/IID/sharedlibs/spring/hamcrest-core-2.2.jar
c:/IBM/IID/sharedlibs/spring/hamcrest-library-2.2.jar
c:/IBM/IID/sharedlibs/spring/http-client-3.8.0.RELEASE.jar
c:/IBM/IID/sharedlibs/spring/jackson-annotations-2.11.4.jar
c:/IBM/IID/sharedlibs/spring/jackson-core-2.11.4.jar
c:/IBM/IID/sharedlibs/spring/jackson-databind-2.11.4.jar
c:/IBM/IID/sharedlibs/spring/jackson-dataformat-xml-2.11.4.jar
c:/IBM/IID/sharedlibs/spring/jackson-datatype-jdk8-2.11.4.jar
c:/IBM/IID/sharedlibs/spring/jackson-datatype-jsr310-2.11.4.jar
c:/IBM/IID/sharedlibs/spring/jackson-module-jaxb-annotations-2.11.4.jar
c:/IBM/IID/sharedlibs/spring/jackson-module-parameter-names-2.11.4.jar
c:/IBM/IID/sharedlibs/spring/jakarta.activation-api-1.2.2.jar
c:/IBM/IID/sharedlibs/spring/jakarta.annotation-api-1.3.5.jar
c:/IBM/IID/sharedlibs/spring/jakarta.el-3.0.3.jar
c:/IBM/IID/sharedlibs/spring/json-path-2.4.0.jar
c:/IBM/IID/sharedlibs/spring/json-smart-2.3.jar
c:/IBM/IID/sharedlibs/spring/jsonassert-1.5.0.jar
c:/IBM/IID/sharedlibs/spring/objenesis-3.1.jar
c:/IBM/IID/sharedlibs/spring/reactive-streams-1.0.3.jar
c:/IBM/IID/sharedlibs/spring/reactor-core-3.4.2.jar
c:/IBM/IID/sharedlibs/spring/snakeyaml-1.27.jar
c:/IBM/IID/sharedlibs/spring/spring-amqp-2.3.4.jar
c:/IBM/IID/sharedlibs/spring/spring-aop-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-beans-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-boot-2.4.2.jar
c:/IBM/IID/sharedlibs/spring/spring-boot-autoconfigure-2.4.2.jar
c:/IBM/IID/sharedlibs/spring/spring-boot-starter-2.4.2.jar
c:/IBM/IID/sharedlibs/spring/spring-boot-starter-amqp-2.4.2.jar
c:/IBM/IID/sharedlibs/spring/spring-boot-starter-json-2.4.2.jar
c:/IBM/IID/sharedlibs/spring/spring-boot-starter-logging-2.4.2.jar
c:/IBM/IID/sharedlibs/spring/spring-boot-starter-web-2.4.2.jar
c:/IBM/IID/sharedlibs/spring/spring-context-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-core-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-expression-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-integration-amqp-5.4.3.jar
c:/IBM/IID/sharedlibs/spring/spring-integration-core-5.4.3.jar
c:/IBM/IID/sharedlibs/spring/spring-integration-jms-5.4.3.jar
c:/IBM/IID/sharedlibs/spring/spring-integration-xml-5.4.3.jar
c:/IBM/IID/sharedlibs/spring/spring-jcl-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-jms-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-messaging-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-oxm-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-rabbit-2.3.4.jar
c:/IBM/IID/sharedlibs/spring/spring-rabbit-junit-2.3.4.jar
c:/IBM/IID/sharedlibs/spring/spring-retry-1.3.1.jar
c:/IBM/IID/sharedlibs/spring/spring-tx-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-web-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-webmvc-5.3.3.jar
c:/IBM/IID/sharedlibs/spring/spring-xml-3.0.10.RELEASE.jar
c:/IBM/IID/sharedlibs/spring/stax2-api-4.2.1.jar
c:/IBM/IID/sharedlibs/spring/woodstox-core-6.2.4.jar
c:/IBM/IID/sharedlibs/spring/xmlunit-core-2.7.0.jar
c:/IBM/IID/sharedlibs/spring/slf4j-api-1.7.30.jar
c:/IBM/IID/sharedlibs/spring/jakarta.validation-api-2.0.2.jar
c:/IBM/IID/sharedlibs/spring/hibernate-validator-6.1.7.Final.jar
c:/IBM/IID/sharedlibs/spring/jboss-logging-3.4.1.Final.jar
c:/IBM/IID/sharedlibs/spring/classmate-1.5.1.jar
c:/IBM/IID/sharedlibs/spring/javax.jms-api-2.0.1.jar
UPDATE
So what I finally realized is that:
WAS 8.5.5 is using J2EE v6, which means JMS 1.1
Spring JMS is using JMS 2.0
When I switched to using the UserCredentialsConnectionFactoryAdapter, this tries to use the JmsContext interface which is part of JMS 2.0 classes, and not provided by the WAS jee container, so this was the reason for the class cast exception.
What I did was to do the JMS sending manually instead of using any spring integration gateway. A better solution might be to create my own adapter that extends connection factory and uses credentials in the connect method, but this works well enough for now:
.handle( m -> {
try {
jmsConfig.sendMessage( m.getPayload().toString() );
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
} )
JmsConfig being a bean that manages the connection.

JPA transactional proxy within a thread issue

In my Controller I have injected (#Autowired) this Service, which implements Runnable (I need multi-threading) and I call it like so:
Thread t = new Thread(service);
t.start();
t.join();
Then, in my Service's run() I call this Repository (simple JPARepository), which is injected in Service also with #Autowired:
repository.save(someEntity);
The problem is that it doesn't persist the entity with id=1. The transactional proxy (and Hibernate connection pool) is initialized after the unsuccessful saving of the first entity. After that, it works fine.
Can anyone point me to the right direction with my issue: how to force the Thread to initialize the Hibernate transactional proxy before persisting the first entity?
You should consider to start the thread after Spring context is refreshed. This is safer because all your beans may be in an inconsistent state.
#EventListener(ContextRefreshedEvent.class)
public void handleContextStart() {
// ...
}

Pre-bound JDBC Connection found

We have an app that is using hibernate, spring, and DB2 in websphere 7. We have audit triggers and we need to set so the triggers can know the logged in user (we use generic logon to the database). We came up with a new scheme for setting this in a new app so that it can automatically join in new transactions. We overrode the transaction manager and did the work in the doBegin.
These scheme worked great in one app, and seemed to work great in a second app, but now, weeks later, and not consistently (behavior is intermittent and does not happen in local development), we are getting this Pre-bound JDBC Connection found error. Looking online most posts say this is when you use two transaction managers against one data source. That is now what we are doing.
I also read one post wondering if it was because he mixed annotation and AOP based transactions. This app does some of that. I don't really buy that theory, but thought I'd mention it.
Exception:
Caused by:
org.springframework.transaction.IllegalTransactionStateException: Pre-bound JDBC Connection found! HibernateTransactionManager does not support running within DataSourceTransactionManager if told to manage the DataSource itself. It is recommended to use a single HibernateTransactionManager for all transactions on a single DataSource, no matter whether Hibernate or JDBC access.
at java.lang.Throwable.<init>(Throwable.java:67)
at org.springframework.core.NestedRuntimeException.<init>(NestedRuntimeException.java:54)
at org.springframework.transaction.TransactionException.<init>(TransactionException.java:34)
at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:475)
at gov.usdoj.afms.umc.utils.hibernate.AfmsHibernateTransactionManager.doBegin(AfmsHibernateTransactionManager.java:28)
Code (note that the exception comes from the super.doBegin()):
protected void doBegin(Object arg0, TransactionDefinition arg1) {
super.doBegin(arg0, arg1);
if (!Db2ClientInfo.exists()) {
clearDBProperty();
} else {
setDBProperty(Db2ClientInfo.getClientUserId(), Db2ClientInfo.getClientApplicationId());
}
}
private void setDBProperty(String uId, String appName) {
Session session = getSessionFactory().getCurrentSession();
Properties props = new Properties();
props.setProperty(WSConnection.CLIENT_ID, uId);
props.setProperty(WSConnection.CLIENT_APPLICATION_NAME, appName);
try {
Connection nativeConn = new SimpleNativeJdbcExtractor().getNativeConnection(session.connection());
if (nativeConn instanceof WSConnection) {
WSConnection wconn = (WSConnection) nativeConn;
wconn.setClientInformation(props);
} else {
logger.error("Connection was NOT an instance of WSConnection so client ID and app could not be set");
}
} catch (Exception e) {
throw new RuntimeException("Cannot set DB parameters!", e);
}
}
I just realized I never answered this. It turns out that the exception had nothing whatever to do with our Tx manager. It was the fact that this particular EAR has two apps in it, each pointing to the same data source. Evidently this confuses hibernate. We've plans to separate the apps some day, but creating an identical (except in name) data source and pointing the apps at them separately fixes the issue for now.
Instead of modifying the transaction manager it might be easier (better?) to create a wrapper around your datasource (extending DelegatingDataSource from spring) and override the 2 getConnection methods. For the cleanup you could wrap the connection in a proxy and intercept the close method.
That should be a safer (and easier I guess) way then trying to fiddle with the transaction manager and it works for every technology (JDBC, Hibernate, JPA etc.) as long as you use the wrapped datasource. (The registration could be done with a BeanPostProcessor which detects DataSource instances and simply wraps them in the delegate).
If that is to radical (as it means changing your current applications instead of updating a library). It could be a configuration problem, make sure that you are only loading your configuration (and thus DataSource and TransactionManager) only once, duplicating bean instances might lead to a similair behavior.
For posterity, I just got this problem and the answers here weren't very helpful. We resolved the problem by removing a double import of a core XML file which had the AOP transaction manager definition in it:
<tx:annotation-driven transaction-manager="..."
proxy-target-class="true" />
I'm thinking that it causes there to be 2 transaction managers overlapping the same namespace. To fix it, we moved the imports around so they were being done once.
Hope this helps someone else.

How to delay spring beans startup?

Having spring application (actually grails app) that runs apache-activemq server as spring bean and couple of apache-camel routes. Application use hibernate to work with database. The problem is simple. Activemq+Camel starts up BEFORE grails injects special methods into hibernate domain objects (actually save/update methods etc). So, if activemq already has some data on startup - camel starts processing messages w/o having grails DAO methods injected. This fails with grails.lang.MissingMethodException. Must delay activemq/camel startup before Grails injects special methods into domain objects.
If all these are defined as spring bean, you can use
<bean id="activeMqBean" depends-on="anotherBean" />
This will make sure anotherBean is initialized before activeMqBean
can you move MQ managment into a plugin? It would increase modularity and if you declare in plugin-descriptor
def loadAfter = ['hibernate']
you should have the desired behavior. Works for JBPM plugin
I am not sure in your case but lazy loading may also help e.g.
<bean id="lazybean" class="com.xxx.YourBean" lazy-init="true">
A lazily-initialized bean indicates to the IoC container to create bean instance when it is first requested. This can help you delay the loading of beans you want.
I know this question is pretty old, but I am now facing the same problem in the year 2015 - and this thread does not offer a solution for me.
I came up with a custom processor bean having a CountDownLatch, which I count down after bootstrapping the application. So the messages will be idled until the app has started fully and its working for me.
/**
* bootstrap latch processor
*/
#Log4j
class BootstrapLatchProcessor implements Processor {
private final CountDownLatch latch = new CountDownLatch(1)
#Override
void process(Exchange exchange) throws Exception {
if(latch.count > 0){
log.info "waiting for bootstrapped # ${exchange.fromEndpoint}"
latch.await()
}
exchange.out = exchange.in
}
/**
* mark the application as bootstrapped
*/
public void setBootstrapped(){
latch.countDown()
}
}
Then use it as a bean in your application and call the method setBootstrapped in your Bootstrap.groovy
Then in your RouteBuilder you put the processor between your endpoint and destination for all routes you expect messages coming in before the app has started:
from("activemq:a.in ").processRef('bootstrapProcessor').to("bean:handlerService?method=handle")

Resources