I have been developing some program with Spring and Redis but I am in stuck :(
What I want
when http request comes to server, then setex myKey, value and ttl.
when myKey is expired, then expired event comes to server and I will make a log.
Source code
#Configuration
public class RedisListenerContainerConfig {
#Bean("redisMyListenerContainer")
public RedisMessageListenerContainer redisLPRInOutCarExpiryListenerContainer(
RedisConnectionFactory redisConnectionFactory,
RedisLPRInOutCarExpiryListener redisLPRInOutCarExpiryListener) {
redisConnectionFactory.getConnection().setConfig("notify-keyspace-events", "Kx");
RedisMessageListenerContainer container = new RedisMessageListenerContainer();
container.setConnectionFactory(redisConnectionFactory);
container.addMessageListener(myListener, new PatternTopic("__keyspace#0__:myKey*"));
container.setErrorHandler(e -> log.error("error"));
return container;
}
}
// here expired event will be got
#Component
public class MyListener implements MessageListener {
#Override
public void onMessage(Message message, byte[] pattern) {
// to make log
}
}
#Service
public class MyService {
...
// here setex is generated
private void setExMyKey(String myKey) {
String value = "value";
redisTemplate.opsForValue().set(myKey, value, 60 * 10 ,TimeUnit.SECONDS);
final String listKey = "list_" + myKey;
if (redisTemplate.opsForList().size(listKey) < 1) {
redisTemplate.opsForList().leftPush(listKey, value);
}
}
}
info of Redis
standalone
keys: 610000
expires: 600000
Problem
Actually, the http requests keep coming to the server so it supposes to the ttl of mykey will never be zero.
But after taking a while, although ttl still remains, expired event is generated so many times.
127.0.0.1:6379> ttl myKey_3378
(integer) 1783
the suspicious part
According to Redis documents, I think, my function for setex does not change the value but only ttl.
And that could mean myKey is not really modified.
IMPORTANT all the commands generate events only if the target key is really modified.
For instance an SREM deleting a non-existing element from a Set will not actually change the value of the key, so no event will be generated.
Is it possible to be a issue of Redis lock?
My language is not English so I am sorry in advance not to understand very well.
Hopefully I solve this problem soon.
Thank you so much.
Related
I have a feign client like this with endpoints to two APIs from PROJECT-SERVICE
#FeignClient(name = "PROJECT-SERVICE", fallbackFactory = ProjectServiceFallbackFactory.class)
public interface ProjectServiceClient {
#GetMapping("/api/projects/{projectKey}")
public ResponseEntity<Project> getProjectDetails(#PathVariable("projectKey") String projectKey);
#PostMapping("/api/projects")
public ResponseEntity<Project> createProject(#RequestBody Project project);
}
I'm using those clients like this:
#Service
public class MyService {
#Autowired
private ProjectServiceClient projectServiceClient;
public void doSomething() {
// Some code
ResponseEntity<Project> projectResponse = projectServiceClient.getProjectDetails(projectKey);
// Some more code
}
public void doSomethingElse() {
// Some code
ResponseEntity<Project> projectResponse = projectServiceClient.createProject(Project projectToBeCreated);
// Some more code
}
}
My problem is, most of the times (around 60% of the time), either one of these Feign calls result in a HystrixTimeoutException.
I initially thought there could be a problem in the downstream micro service (PROJECT-SERVICE in this case), but that is not the case. In fact, when getProjectDetails() or createProject() is called, the PROJECT-SERVICE actually does the job and returns a ResponseEntity<Project> with status 200 and 201 respectively, but my fallback is activated with the HystrixTimeoutException.
I'm trying in vain to find what might be causing this issue.
I, however, have this in my main application configuration:
feign.hystrix.enabled=true
feign.client.config.default.connect-timeout=5000
feign.client.config.default.read-timeout=60000
Can anyone point me towards a solution?
Thanks,
Sriram Sridharan
Hystrix's timeout is not tied to that of Feign. There is a default 1 second execution timeout enabled for Hystrix. You need to configure this timeout to be slightly longer than Feign's, to avoid HystrixTimeoutException getting thrown earlier than desired timeout. Like so:
feign.client.config.default.connect-timeout=5000
feign.client.config.default.read-timeout=5000
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds=6000
Doing so would allow FeignException, caused by timeout after 5 seconds, to be thrown first, and then wrapped in a HystrixTimeoutException
I am trying to cache Kafka Records within 3 minutes of interval post that it will get expired and removed from the cache.
Each incoming records which is fetched using kafka consumer written in springboot needs to be updated in cache first then if it is present i need to discard the next duplicate records if it matches the cache record.
I have tried using Caffeine cache as below,
#EnableCaching
public class AppCacheManagerConfig {
#Bean
public CacheManager cacheManager(Ticker ticker) {
CaffeineCache bookCache = buildCache("declineRecords", ticker, 3);
SimpleCacheManager cacheManager = new SimpleCacheManager();
cacheManager.setCaches(Collections.singletonList(bookCache));
return cacheManager;
}
private CaffeineCache buildCache(String name, Ticker ticker, int minutesToExpire) {
return new CaffeineCache(name, Caffeine.newBuilder().expireAfterWrite(minutesToExpire, TimeUnit.MINUTES)
.maximumSize(100).ticker(ticker).build());
}
#Bean
public Ticker ticker() {
return Ticker.systemTicker();
}
}
and my Kafka Consumer is as below,
#Autowired
CachingServiceImpl cachingService;
#KafkaListener(topics = "#{'${spring.kafka.consumer.topic}'}", concurrency = "#{'${spring.kafka.consumer.concurrentConsumers}'}", errorHandler = "#{'${spring.kafka.consumer.errorHandler}'}")
public void consume(Message<?> message, Acknowledgment acknowledgment,
#Header(KafkaHeaders.RECEIVED_TIMESTAMP) long createTime) {
logger.info("Recieved Message: " + message.getPayload());
try {
boolean approveTopic = false;
boolean duplicateRecord = false;
if (cachingService.isDuplicateCheck(declineRecord)) {
//do something with records
}
else
{
//do something with records
}
cachingService.putInCache(xmlJSONObj, declineRecord, time);
and my caching service is as below,
#Component
public class CachingServiceImpl {
private static final Logger logger = LoggerFactory.getLogger(CachingServiceImpl.class);
#Autowired
CacheManager cacheManager;
#Cacheable(value = "declineRecords", key = "#declineRecord", sync = true)
public String putInCache(JSONObject xmlJSONObj, String declineRecord, String time) {
logger.info("Record is Cached for 3 minutes interval check", declineRecord);
cacheManager.getCache("declineRecords").put(declineRecord, time);
return declineRecord;
}
public boolean isDuplicateCheck(String declineRecord) {
if (null != cacheManager.getCache("declineRecords").get(declineRecord)) {
return true;
}
return false;
}
}
But Each time a record comes in consumer my cache is always empty. Its not holding the records.
Modifications Done:
I have added Configuration file as below after going through the suggestions and more kind of R&D removed some of the earlier logic and now the caching is working as expected but duplicate check is failing when all the three consumers are sending the same records.
`
#Configuration
public class AppCacheManagerConfig {
public static Cache<String, Object> jsonCache =
Caffeine.newBuilder().expireAfterWrite(3, TimeUnit.MINUTES)
.maximumSize(10000).recordStats().build();
#Bean
public CacheLoader<Object, Object> cacheLoader() {
CacheLoader<Object, Object> cacheLoader = new CacheLoader<Object, Object>() {
#Override
public Object load(Object key) throws Exception {
return null;
}
#Override
public Object reload(Object key, Object oldValue) throws Exception {
return oldValue;
}
};
return cacheLoader;
}
`
Now i am using the above cache as manual put and get.
I guess you're trying to implement records deduplication for Kafka.
Here is the similar discussion:
https://github.com/spring-projects/spring-kafka/issues/80
Here is the current abstract class which you may extend to achieve the necessary result:
https://github.com/spring-projects/spring-kafka/blob/master/spring-kafka/src/main/java/org/springframework/kafka/listener/adapter/AbstractFilteringMessageListener.java
Your caching service is definitely incorrect: Cacheable annotation allows marking the data getters and setters, to add caching through AOP. While in the code you clearly implement some low-level cache updating logic of your own.
At least next possible changes may help you:
Remove #Cacheable. You don't need it because you work with cache manually, so it may be the source of conflicts (especially as soon as you use sync = true). If it helps, remove #EnableCaching as well - it enables support for cache-related Spring annotations which you don't need here.
Try removing Ticker bean with the appropriate parameters for other beans. It should not be harmful as per your configuration, but usually it's helpful only for tests, no need to define it otherwise.
Double-check what is declineRecord. If it's a serialized object, ensure that serialization works properly.
Add recordStats() for cache and output stats() to log for further analysis.
I've been trying to integrate Hazelcast into my application but am running into a behaviour I had not anticipated with the onExpired vs onRemoved listener.
Ideally, I would like to execute some code whenever a value is removed from my cache. I configured an Expiry policy on the cache, and am expecting that my onRemoved listener will follow after my cache value expires, but it does not seem to be the case.
Does Hazelcast call the onRemoved listener after when it removes an expired value from the cache, or only on an explicit cache.remove() call?
My configuration is:
hazelcastInstance = HazelcastInstanceFactory.getOrCreateHazelcastInstance(getHazelcastConfig());
// Add cache used by adams
CacheSimpleConfig cacheSimpleConfig = new CacheSimpleConfig()
.setName(CACHE_NAME)
.setKeyType(UserRolesCacheKey.class.getName())
.setValueType((new String[0]).getClass().getName())
.setReadThrough(true)
.setInMemoryFormat(InMemoryFormat.OBJECT)
.setEvictionConfig(new EvictionConfig()
.setEvictionPolicy(EvictionPolicy.LRU)
.setSize(1000)
.setMaximumSizePolicy(EvictionConfig.MaxSizePolicy.ENTRY_COUNT))
.setExpiryPolicyFactoryConfig(
new ExpiryPolicyFactoryConfig(
new TimedExpiryPolicyFactoryConfig(ACCESSED,
new DurationConfig(
120,
TimeUnit.SECONDS))));
hazelcastInstance.getConfig().addCacheConfig(cacheSimpleConfig);
ICache<UserRolesCacheKey, String[]> userRolesCache = hazelcastInstance.getCacheManager().getCache(CACHE_NAME);
userRolesCache.registerCacheEntryListener(new MutableCacheEntryListenerConfiguration<>(
new UserRolesCacheListenerFactory(), null, false, false));
}
}
}
My Listener is fairly simple:
public class UserRolesCacheListenerFactory implements Factory<CacheEntryListener<UserRolesCacheKey, String[]>> {
#Override
public CacheEntryListener create() {
return new UserRolesCacheEntryListener();
}
}
And:
public class UserRolesCacheEntryListener implements CacheEntryRemovedListener<UserRolesCacheKey, String[]>{
private final static Logger LOG = LoggerFactory.getLogger(UserRolesCacheEntryListener.class);
#Override
public void onRemoved(Iterable<CacheEntryEvent<? extends UserRolesCacheKey, ? extends String[]>> cacheEntryEvents) throws CacheEntryListenerException {
cacheEntryEvents.forEach(this::deleteDBData);
}
I would expect that sometime after 120s my onRemoved method would be called by Hazelcast as it removes the expired value from the cache, but it never seems to be.
Is this expected behaviour? Is something missing in my cache configuration?
According to the JCache specification, section 8.4, the REMOVED event is only for explicit operations.
Listening for EXPIRED event will be better but still not ideal.
Note the wording in the specification and the code here. EXPIRED events are implementation dependent -- a caching provider is allowed to never notice the data has expired, never remove it, and so never generate the event.
Hazelcast does notice see here, but this makes the timely appearance of the event you need dependent on the implementation.
So, I was looking at caching methods in Java (Spring). And Guava looked like it would solve the purpose.
This is the usecase -
I query for some data from a remote service. Kind of configuration field for my application. This field will be used by every inbound request to my application. And it would be expensive to call the remote service everytime as it's kind of constant which changes periodically.
So, on the first request inbound to my application, when I call remote service, I would cache the value. I set an expiry time of this cache as 30 mins. After 30 mins when the cache is expired and there is a request to retrieve the key, I would like a callback or something to do the operation of calling the remote service and setting the cache and return the value for that key.
How can I do it in Guava cache?
Here i give a example how to use guava cache. If you want to handle removal listener then need to call cleanUp. Here i run a thread which one call clean up every 30 minutes.
import com.google.common.cache.*;
import org.springframework.stereotype.Component;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
#Component
public class Cache {
public static LoadingCache<String, String> REQUIRED_CACHE;
public Cache(){
RemovalListener<String,String> REMOVAL_LISTENER = new RemovalListener<String, String>() {
#Override
public void onRemoval(RemovalNotification<String, String> notification) {
if(notification.getCause() == RemovalCause.EXPIRED){
//do as per your requirement
}
}
};
CacheLoader<String,String> LOADER = new CacheLoader<String, String>() {
#Override
public String load(String key) throws Exception {
return null; // return as per your requirement. if key value is not found
}
};
REQUIRED_CACHE = CacheBuilder.newBuilder().maximumSize(100000000)
.expireAfterWrite(30, TimeUnit.MINUTES)
.removalListener(REMOVAL_LISTENER)
.build(LOADER);
Executors.newSingleThreadExecutor().submit(()->{
while (true) {
REQUIRED_CACHE.cleanUp(); // need to call clean up for removal listener
TimeUnit.MINUTES.sleep(30L);
}
});
}
}
put & get data:
Cache.REQUIRED_CACHE.get("key");
Cache.REQUIRED_CACHE.put("key","value");
I have a CXF client configured in my Spring Boot app like so:
#Bean
public ConsumerSupportService consumerSupportService() {
JaxWsProxyFactoryBean jaxWsProxyFactoryBean = new JaxWsProxyFactoryBean();
jaxWsProxyFactoryBean.setServiceClass(ConsumerSupportService.class);
jaxWsProxyFactoryBean.setAddress("https://www.someservice.com/service?wsdl");
jaxWsProxyFactoryBean.setBindingId(SOAPBinding.SOAP12HTTP_BINDING);
WSAddressingFeature wsAddressingFeature = new WSAddressingFeature();
wsAddressingFeature.setAddressingRequired(true);
jaxWsProxyFactoryBean.getFeatures().add(wsAddressingFeature);
ConsumerSupportService service = (ConsumerSupportService) jaxWsProxyFactoryBean.create();
Client client = ClientProxy.getClient(service);
AddressingProperties addressingProperties = new AddressingProperties();
AttributedURIType to = new AttributedURIType();
to.setValue(applicationProperties.getWex().getServices().getConsumersupport().getTo());
addressingProperties.setTo(to);
AttributedURIType action = new AttributedURIType();
action.setValue("http://serviceaction/SearchConsumer");
addressingProperties.setAction(action);
client.getRequestContext().put("javax.xml.ws.addressing.context", addressingProperties);
setClientTimeout(client);
return service;
}
private void setClientTimeout(Client client) {
HTTPConduit conduit = (HTTPConduit) client.getConduit();
HTTPClientPolicy policy = new HTTPClientPolicy();
policy.setConnectionTimeout(applicationProperties.getWex().getServices().getClient().getConnectionTimeout());
policy.setReceiveTimeout(applicationProperties.getWex().getServices().getClient().getReceiveTimeout());
conduit.setClient(policy);
}
This same service bean is accessed by two different threads in the same application sequence. If I execute this particular sequence 10 times in a row, I will get a connection timeout from the service call at least 3 times. What I'm seeing is:
Caused by: java.io.IOException: Timed out waiting for response to operation {http://theservice.com}SearchConsumer.
at org.apache.cxf.endpoint.ClientImpl.waitResponse(ClientImpl.java:685) ~[cxf-core-3.2.0.jar:3.2.0]
at org.apache.cxf.endpoint.ClientImpl.processResult(ClientImpl.java:608) ~[cxf-core-3.2.0.jar:3.2.0]
If I change the sequence such that one of the threads does not call this service, then the error goes away. So, it seems like there's some sort of a race condition happening here. If I look at the logs in our proxy manager for this service, I can see that both of the service calls do return a response very quickly, but the second service call seems to get stuck somewhere in the code and never actually lets go of the connection until the timeout value is reached. I've been trying to track down the cause of this for quite a while, but have been unsuccessful.
I've read some mixed opinions as to whether or not CXF client proxies are thread-safe, but I was under the impression that they were. If this actually not the case, and I should be creating a new client proxy for each invocation, or use a pool of proxies?
Turns out that it is an issue with the proxy not being thread-safe. What I wound up doing was leveraging a solution kind of like one posted at the bottom of this post: Is this JAX-WS client call thread safe? - I created a pool for the proxies and I use that to access proxies from multiple threads in a thread-safe manner. This seems to work out pretty well.
public class JaxWSServiceProxyPool<T> extends GenericObjectPool<T> {
JaxWSServiceProxyPool(Supplier<T> factory, GenericObjectPoolConfig poolConfig) {
super(new BasePooledObjectFactory<T>() {
#Override
public T create() throws Exception {
return factory.get();
}
#Override
public PooledObject<T> wrap(T t) {
return new DefaultPooledObject<>(t);
}
}, poolConfig != null ? poolConfig : new GenericObjectPoolConfig());
}
}
I then created a simple "registry" class to keep references to various pools.
#Component
public class JaxWSServiceProxyPoolRegistry {
private static final Map<Class, JaxWSServiceProxyPool> registry = new HashMap<>();
public synchronized <T> void register(Class<T> serviceTypeClass, Supplier<T> factory, GenericObjectPoolConfig poolConfig) {
Assert.notNull(serviceTypeClass);
Assert.notNull(factory);
if (!registry.containsKey(serviceTypeClass)) {
registry.put(serviceTypeClass, new JaxWSServiceProxyPool<>(factory, poolConfig));
}
}
public <T> void register(Class<T> serviceTypeClass, Supplier<T> factory) {
register(serviceTypeClass, factory, null);
}
#SuppressWarnings("unchecked")
public <T> JaxWSServiceProxyPool<T> getServiceProxyPool(Class<T> serviceTypeClass) {
Assert.notNull(serviceTypeClass);
return registry.get(serviceTypeClass);
}
}
To use it, I did:
JaxWSServiceProxyPoolRegistry jaxWSServiceProxyPoolRegistry = new JaxWSServiceProxyPoolRegistry();
jaxWSServiceProxyPoolRegistry.register(ConsumerSupportService.class,
this::buildConsumerSupportServiceClient,
getConsumerSupportServicePoolConfig());
Where buildConsumerSupportServiceClient uses a JaxWsProxyFactoryBean to build up the client.
To retrieve an instance from the pool I inject my registry class and then do:
JaxWSServiceProxyPool<ConsumerSupportService> consumerSupportServiceJaxWSServiceProxyPool = jaxWSServiceProxyPoolRegistry.getServiceProxyPool(ConsumerSupportService.class);
And then borrow/return the object from/to the pool as necessary.
This seems to work well so far. I've executed some fairly heavy load tests against it and it's held up.