I am new to caching and Spring, I can't work out the difference between cacheNames and Key in below example taken from Spring Docs:
#Cacheable(cacheNames="books", key="#isbn")
public Book findBook(ISBN isbn, boolean checkWarehouse, boolean includeUsed)
As I understand cache is simply a key-value pair stored in memory. So in the above example on first invocation the returned Book value will be stored in cache using the value of isbn parameter as key. On subsequent invocations where isbn value is the same as it was first requested the Book stored in cache will be returned. This Book in cache will be found using the Key. So what is cacheNames?
Am I correct in saying cache is stored as key values like this:
isbn111111 ---> Book,
isbn122222 ---> Book2,
isbn123333 ---> Book3
Thanks in advance.
CacheName is more like group of cache key. When you open this class
org.springframework.cache.interceptor.AbstractCacheResolver
you will find this method to find cache by cacheName
#Override
public Collection<? extends Cache> resolveCaches(CacheOperationInvocationContext<?> context) {
Collection<String> cacheNames = getCacheNames(context);
if (cacheNames == null) {
return Collections.emptyList();
}
Collection<Cache> result = new ArrayList<>(cacheNames.size());
for (String cacheName : cacheNames) {
Cache cache = getCacheManager().getCache(cacheName);
if (cache == null) {
throw new IllegalArgumentException("Cannot find cache named '" +
cacheName + "' for " + context.getOperation());
}
result.add(cache);
}
return result;
}
So later in org.springframework.cache.interceptor.CacheAspectSupport spring will get value by cache key from that cache object
private Object execute(final CacheOperationInvoker invoker, Method method, CacheOperationContexts contexts) {
// Special handling of synchronized invocation
if (contexts.isSynchronized()) {
CacheOperationContext context = contexts.get(CacheableOperation.class).iterator().next();
if (isConditionPassing(context, CacheOperationExpressionEvaluator.NO_RESULT)) {
Object key = generateKey(context, CacheOperationExpressionEvaluator.NO_RESULT);
Cache cache = context.getCaches().iterator().next();
try {
return wrapCacheValue(method, cache.get(key, () -> unwrapReturnValue(invokeOperation(invoker))));
}
catch (Cache.ValueRetrievalException ex) {
// The invoker wraps any Throwable in a ThrowableWrapper instance so we
// can just make sure that one bubbles up the stack.
throw (CacheOperationInvoker.ThrowableWrapper) ex.getCause();
}
}
//...other logic
The cacheNames are the names of the caches itself, where the data is stored. You can have multiple caches, e.g. for different entity types different caches, or depending on replication needs etc.
One significance of cacheNames would be helping with default key generation for #Cacheable used when explicit keys aren't passed to method. Its very unclear from Spring documentation on what would be seriously wrong or inaccurate if cacheNames is not supplied at Class level or Method level when using Spring Cache.
https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/cache/annotation/CacheConfig.html#cacheNames--
Related
I am trying to cache Kafka Records within 3 minutes of interval post that it will get expired and removed from the cache.
Each incoming records which is fetched using kafka consumer written in springboot needs to be updated in cache first then if it is present i need to discard the next duplicate records if it matches the cache record.
I have tried using Caffeine cache as below,
#EnableCaching
public class AppCacheManagerConfig {
#Bean
public CacheManager cacheManager(Ticker ticker) {
CaffeineCache bookCache = buildCache("declineRecords", ticker, 3);
SimpleCacheManager cacheManager = new SimpleCacheManager();
cacheManager.setCaches(Collections.singletonList(bookCache));
return cacheManager;
}
private CaffeineCache buildCache(String name, Ticker ticker, int minutesToExpire) {
return new CaffeineCache(name, Caffeine.newBuilder().expireAfterWrite(minutesToExpire, TimeUnit.MINUTES)
.maximumSize(100).ticker(ticker).build());
}
#Bean
public Ticker ticker() {
return Ticker.systemTicker();
}
}
and my Kafka Consumer is as below,
#Autowired
CachingServiceImpl cachingService;
#KafkaListener(topics = "#{'${spring.kafka.consumer.topic}'}", concurrency = "#{'${spring.kafka.consumer.concurrentConsumers}'}", errorHandler = "#{'${spring.kafka.consumer.errorHandler}'}")
public void consume(Message<?> message, Acknowledgment acknowledgment,
#Header(KafkaHeaders.RECEIVED_TIMESTAMP) long createTime) {
logger.info("Recieved Message: " + message.getPayload());
try {
boolean approveTopic = false;
boolean duplicateRecord = false;
if (cachingService.isDuplicateCheck(declineRecord)) {
//do something with records
}
else
{
//do something with records
}
cachingService.putInCache(xmlJSONObj, declineRecord, time);
and my caching service is as below,
#Component
public class CachingServiceImpl {
private static final Logger logger = LoggerFactory.getLogger(CachingServiceImpl.class);
#Autowired
CacheManager cacheManager;
#Cacheable(value = "declineRecords", key = "#declineRecord", sync = true)
public String putInCache(JSONObject xmlJSONObj, String declineRecord, String time) {
logger.info("Record is Cached for 3 minutes interval check", declineRecord);
cacheManager.getCache("declineRecords").put(declineRecord, time);
return declineRecord;
}
public boolean isDuplicateCheck(String declineRecord) {
if (null != cacheManager.getCache("declineRecords").get(declineRecord)) {
return true;
}
return false;
}
}
But Each time a record comes in consumer my cache is always empty. Its not holding the records.
Modifications Done:
I have added Configuration file as below after going through the suggestions and more kind of R&D removed some of the earlier logic and now the caching is working as expected but duplicate check is failing when all the three consumers are sending the same records.
`
#Configuration
public class AppCacheManagerConfig {
public static Cache<String, Object> jsonCache =
Caffeine.newBuilder().expireAfterWrite(3, TimeUnit.MINUTES)
.maximumSize(10000).recordStats().build();
#Bean
public CacheLoader<Object, Object> cacheLoader() {
CacheLoader<Object, Object> cacheLoader = new CacheLoader<Object, Object>() {
#Override
public Object load(Object key) throws Exception {
return null;
}
#Override
public Object reload(Object key, Object oldValue) throws Exception {
return oldValue;
}
};
return cacheLoader;
}
`
Now i am using the above cache as manual put and get.
I guess you're trying to implement records deduplication for Kafka.
Here is the similar discussion:
https://github.com/spring-projects/spring-kafka/issues/80
Here is the current abstract class which you may extend to achieve the necessary result:
https://github.com/spring-projects/spring-kafka/blob/master/spring-kafka/src/main/java/org/springframework/kafka/listener/adapter/AbstractFilteringMessageListener.java
Your caching service is definitely incorrect: Cacheable annotation allows marking the data getters and setters, to add caching through AOP. While in the code you clearly implement some low-level cache updating logic of your own.
At least next possible changes may help you:
Remove #Cacheable. You don't need it because you work with cache manually, so it may be the source of conflicts (especially as soon as you use sync = true). If it helps, remove #EnableCaching as well - it enables support for cache-related Spring annotations which you don't need here.
Try removing Ticker bean with the appropriate parameters for other beans. It should not be harmful as per your configuration, but usually it's helpful only for tests, no need to define it otherwise.
Double-check what is declineRecord. If it's a serialized object, ensure that serialization works properly.
Add recordStats() for cache and output stats() to log for further analysis.
I implemented a non-blocking cache using Google Guava, there's only one key in the cache, and value for the key is only refreshed asynchronously (by overriding reload()).
My question is that does Guava cache handle de-duplication if the first reload() task hasn't finished, and a new get() request comes in.
//Cache is defined like below
this.cache = CacheBuilder
.newBuilder()
.maximumSize(1)
.refreshAfterWrite(10, TimeUnit.MINUTES)
.recordStats()
.build(loader);
//reload is overwritten asynchronously
#Override
public ListenableFuture<Map<String, CertificateInfo>> reload(final String key, Map<String, CertificateInfo> prevMap) throws IOException {
LOGGER.info("Refreshing certificate cache.");
ListenableFutureTask<Map<String, CertificateInfo>> task = ListenableFutureTask.create(new Callable<Map<String, CertificateInfo>>() {
#Override
public Map<String, CertificateInfo> call() throws Exception {
return actuallyLoad();
}
});
executor.execute(task);
return task;
}
Yes, see the documentation for LoadingCache.get(K) (and it sibling, Cache.get(K, Runnable)):
If another call to get(K) or getUnchecked(K) is currently loading the value for key, simply waits for that thread to finish and returns its loaded value.
So if a cache entry is currently being computed (or reloaded/recomputed), other threads that try to retrieve that entry will simply wait for the computation to finish - they will not kick off their own redundant refresh.
I am working with Spring and EhCache
I have the following method
#Override
#Cacheable(value="products", key="#root.target.PRODUCTS")
public Set<Product> findAll() {
return new LinkedHashSet<>(this.productRepository.findAll());
}
I have other methods working with #Cacheable and #CachePut and #CacheEvict.
Now, imagine the database returns 100 products and they are cached through key="#root.target.PRODUCTS", then other method would insert - update - deleted an item into the database. Therefore the products cached through the key="#root.target.PRODUCTS" are not the same anymore such as the database.
I mean, check the two following two methods, they are able to update/delete an item, and that same item is cached in the other key="#root.target.PRODUCTS"
#Override
#CachePut(value="products", key="#product.id")
public Product update(Product product) {
return this.productRepository.save(product);
}
#Override
#CacheEvict(value="products", key="#id")
public void delete(Integer id) {
this.productRepository.delete(id);
}
I want to know if is possible update/delete the item located in the cache through the key="#root.target.PRODUCTS", it would be 100 with the Product updated or 499 if the Product was deleted.
My point is, I want avoid the following:
#Override
#CachePut(value="products", key="#product.id")
#CacheEvict(value="products", key="#root.target.PRODUCTS")
public Product update(Product product) {
return this.productRepository.save(product);
}
#Override
#Caching(evict={
#CacheEvict(value="products", key="#id"),
#CacheEvict(value="products", key="#root.target.PRODUCTS")
})
public void delete(Integer id) {
this.productRepository.delete(id);
}
I don't want call again the 500 or 499 products to be cached into the key="#root.target.PRODUCTS"
Is possible do this? How?
Thanks in advance.
Caching the collection using the caching abstraction is a duplicate of what the underlying caching system is doing. And because this is a duplicate, it turns out that you have to resort to some kind of duplications in your own code in one way or the other (the duplicate key for the set is the obvious representation of that). And because there is duplication, you have to sync state somehow
If you really need to access to the whole set and individual elements, then you should probably use a shortcut for the easiest leg. First, you should make sure your cache contains all elements which is not something that is obvious. Far from it actually. Considering you have that:
//EhCacheCache cache = (EhCacheCache) cacheManager.getCache("products");
#Override
public Set<Product> findAll() {
Ehcache nativeCache = cache.getNativeCache();
Map<Object, Element> elements = nativeCache.getAll(nativeCache.getKeys());
Set<Product> result = new HashSet<Product>();
for (Element element : elements.values()) {
result.add((Product) element.getObjectValue());
}
return Collections.unmodifiableSet(result);
}
The elements result is actually a lazy loaded map so a call to values() may throw an exception. You may want to loop over the keys or something.
You have to remember that the caching abstraction eases the access to the underlying caching infrastructure and in no way it replaces it: if you had to use the API directly, this is probably what you would have to do in some sort.
Now, we can keep the conversion on SPR-12036 if you believe we can improve the caching abstraction in that area. Thanks!
I think something like this schould work... Actually it's only a variation if "Stéphane Nicoll" answer ofcourse, but it may be useful for someone. I write it right here and haven't check it in IDE, but something similar works in my Project.
Override CacheResolver:
#Cacheable(value="products", key="#root.target.PRODUCTS", cacheResolver = "customCacheResolver")
Implement your own cache resolver, which search "inside" you cached items and do the work in there
public class CustomCacheResolver implements CacheResolver{
private static final String CACHE_NAME = "products";
#Autowired(required = true) private CacheManager cacheManager;
#SuppressWarnings("unchecked")
#Override
public Collection<? extends Cache> resolveCaches(CacheOperationInvocationContext<?> cacheOperationInvocationContext) {
// 1. Take key from query and create new simple key
SimpleKey newKey;
if (cacheOperationInvocationContext.getArgs().length != null) { //optional
newKey = new SimpleKey(args); //It's the key of cached object, which your "#Cachable" search for
} else {
//Schould never be... DEFAULT work with cache if something wrong with arguments
return new ArrayList<>(Arrays.asList(cacheManager.getCache(CACHE_NAME)));
}
// 2. Take cache
EhCacheCache ehCache = (EhCacheCache)cacheManager.getCache(CACHE_NAME); //this one we bringing back
Ehcache cache = (Ehcache)ehCache.getNativeCache(); //and with this we working
// 3. Modify existing Cache if we have it
if (cache.getKeys().contains(newKey) && YouWantToModifyIt) {
Element element = cache.get(key);
if (element != null && !((List<Products>)element.getObjectValue()).isEmpty()) {
List<Products> productsList = (List<Products>)element.getObjectValue();
// ---**--- Modify your "productsList" here as you want. You may now Change single element in this list.
ehCache.put(key, anfragenList); //this method NOT adds cache, but OVERWRITE existing
// 4. Maybe "Create" new cache with this key if we don't have it
} else {
ehCache.put(newKey, YOUR_ELEMENTS);
}
return new ArrayList<>(Arrays.asList(ehCache)); //Bring all back - our "new" or "modified" cache is in there now...
}
Read more about CRUD of EhCache: EhCache code samples
Hope it helps. And sorry for my English:(
I think there is a way to read the collection from underlying cache structure of spring. You can retrieve the collection from underlying ConcurrentHashMap as key-value pairs without using EhCache or anything else. Then you can update or remove an entry from that collection and then you can update the cache too. Here is an example that may help:
import com.crud.model.Post;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cache.Cache;
import org.springframework.cache.CacheManager;
import org.springframework.cache.interceptor.CacheOperationInvocationContext;
import org.springframework.cache.interceptor.CacheResolver;
import org.springframework.cache.interceptor.SimpleKey;
import org.springframework.stereotype.Component;
import java.util.*;
#Component
#Slf4j
public class CustomCacheResolver implements CacheResolver {
private static final String CACHE_NAME = "allPost";
#Autowired
private CacheManager cacheManager;
#SuppressWarnings("unchecked")
#Override
public Collection<? extends Cache> resolveCaches(CacheOperationInvocationContext<?> cacheOperationInvocationContext) {
log.info(Arrays.toString(cacheOperationInvocationContext.getArgs()));
String method = cacheOperationInvocationContext.getMethod().toString();
Post post = null;
Long postId = null;
if(method.contains("update")) {
//get the updated post
Object[] args = cacheOperationInvocationContext.getArgs();
post = (Post) args[0];
}
else if(method.contains("delete")){
//get the post Id to delete
Object[] args = cacheOperationInvocationContext.getArgs();
postId = (Long) args[0];
}
//read the cache
Cache cache = cacheManager.getCache(CACHE_NAME);
//get the concurrent cache map in key-value pair
assert cache != null;
Map<SimpleKey, List<Post>> map = (Map<SimpleKey, List<Post>>) cache.getNativeCache();
//Convert to set to iterate
Set<Map.Entry<SimpleKey, List<Post>>> entrySet = map.entrySet();
Iterator<Map.Entry<SimpleKey, List<Post>>> itr = entrySet.iterator();
//if a iterated entry is a list then it is our desired data list!!! Yayyy
Map.Entry<SimpleKey, List<Post>> entry = null;
while (itr.hasNext()){
entry = itr.next();
if(entry instanceof List) break;
}
//get the list
assert entry != null;
List<Post> postList = entry.getValue();
if(method.contains("update")) {
//update it
for (Post temp : postList) {
assert post != null;
if (temp.getId().equals(post.getId())) {
postList.remove(temp);
break;
}
}
postList.add(post);
}
else if(method.contains("delete")){
//delete it
for (Post temp : postList) {
if (temp.getId().equals(postId)) {
postList.remove(temp);
break;
}
}
}
//update the cache!! :D
cache.put(entry.getKey(),postList);
return new ArrayList<>(Collections.singletonList(cacheManager.getCache(CACHE_NAME)));
}
}
Here are the methods that uses the CustomCacheResolver
#Cacheable(key = "{#pageNo,#pageSize}")
public List<Post> retrieveAllPost(int pageNo,int pageSize){ // return list}
#CachePut(key = "#post.id",cacheResolver = "customCacheResolver")
public Boolean updatePost(Post post, UserDetails userDetails){ //your logic}
#CachePut(key = "#postId",cacheResolver = "customCacheResolver")
public Boolean deletePost(Long postId,UserDetails userDetails){ // your logic}
#CacheEvict(allEntries = true)
public Boolean createPost(String userId, Post post){//your logic}
Hope it helps to manipulate your spring application cache manually!
Though I don't see any easy way, but you can override Ehcache cache functionality by supplying cache decorator. Most probably you'd want to use EhcahceDecoratorAdapter, to enhance functions used by EhCacheCache put and evict methods.
Simple and rude solution is :
#Cacheable(key = "{#pageNo,#pageSize}")
public List<Post> retrieveAllPost(int pageNo,int pageSize){ // return list}
#CacheEvict(allEntries = true)
public Boolean updatePost(Post post, UserDetails userDetails){ //your logic}
#CacheEvict(allEntries = true)
public Boolean deletePost(Long postId,UserDetails userDetails){ // your logic}
#CacheEvict(allEntries = true)
public Boolean createPost(String userId, Post post){//your logic}
I am using annotations to configure my spring environment like this:
#Configuration
...
#PropertySource("classpath:/config/default.properties")
...
public class GeneralApplicationConfiguration implements WebApplicationInitializer
{
#Autowired
Environment env;
}
This leads to my properties from default.properties being part of the Environment. I want to use the #PropertySource mechanism here, because it already provides the possibility to overload properties through several fallback layers and different dynamic locations, based on the environment settings (e.g. config_dir location). I just stripped the fallback to make the example easier.
However, my problem now is that I want to configure for example my datasource properties in default.properties. You can pass the settings to the datasource without knowing in detail what settings the datasource expects using
Properties p = ...
datasource.setProperties(p);
However, the problem is, the Environment object is neither a Properties object nor a Map nor anything comparable. From my point of view it is simply not possible to access all values of the environment, because there is no keySet or iterator method or anything comparable.
Properties p <=== Environment env?
Am I missing something? Is it possible to access all entries of the Environment object somehow? If yes, I could map the entries to a Map or Properties object, I could even filter or map them by prefix - create subsets as a standard java Map ... This is what I would like to do. Any suggestions?
You need something like this, maybe it can be improved. This is a first attempt:
...
import org.springframework.core.env.PropertySource;
import org.springframework.core.env.AbstractEnvironment;
import org.springframework.core.env.Environment;
import org.springframework.core.env.MapPropertySource;
...
#Configuration
...
#org.springframework.context.annotation.PropertySource("classpath:/config/default.properties")
...
public class GeneralApplicationConfiguration implements WebApplicationInitializer
{
#Autowired
Environment env;
public void someMethod() {
...
Map<String, Object> map = new HashMap();
for(Iterator it = ((AbstractEnvironment) env).getPropertySources().iterator(); it.hasNext(); ) {
PropertySource propertySource = (PropertySource) it.next();
if (propertySource instanceof MapPropertySource) {
map.putAll(((MapPropertySource) propertySource).getSource());
}
}
...
}
...
Basically, everything from the Environment that's a MapPropertySource (and there are quite a lot of implementations) can be accessed as a Map of properties.
This is an old question, but the accepted answer has a serious flaw. If the Spring Environment object contains any overriding values (as described in Externalized Configuration), there is no guarantee that the map of property values it produces will match those returned from the Environment object. I found that simply iterating through the PropertySources of the Environment did not, in fact, give any overriding values. Instead it produced the original value, the one that should have been overridden.
Here is a better solution. This uses the EnumerablePropertySources of the Environment to iterate through the known property names, but then reads the actual value out of the real Spring environment. This guarantees that the value is the one actually resolved by Spring, including any overriding values.
Properties props = new Properties();
MutablePropertySources propSrcs = ((AbstractEnvironment) springEnv).getPropertySources();
StreamSupport.stream(propSrcs.spliterator(), false)
.filter(ps -> ps instanceof EnumerablePropertySource)
.map(ps -> ((EnumerablePropertySource) ps).getPropertyNames())
.flatMap(Arrays::<String>stream)
.forEach(propName -> props.setProperty(propName, springEnv.getProperty(propName)));
I had the requirement to retrieve all properties whose key starts with a distinct prefix (e.g. all properties starting with "log4j.appender.") and wrote following Code (using streams and lamdas of Java 8).
public static Map<String,Object> getPropertiesStartingWith( ConfigurableEnvironment aEnv,
String aKeyPrefix )
{
Map<String,Object> result = new HashMap<>();
Map<String,Object> map = getAllProperties( aEnv );
for (Entry<String, Object> entry : map.entrySet())
{
String key = entry.getKey();
if ( key.startsWith( aKeyPrefix ) )
{
result.put( key, entry.getValue() );
}
}
return result;
}
public static Map<String,Object> getAllProperties( ConfigurableEnvironment aEnv )
{
Map<String,Object> result = new HashMap<>();
aEnv.getPropertySources().forEach( ps -> addAll( result, getAllProperties( ps ) ) );
return result;
}
public static Map<String,Object> getAllProperties( PropertySource<?> aPropSource )
{
Map<String,Object> result = new HashMap<>();
if ( aPropSource instanceof CompositePropertySource)
{
CompositePropertySource cps = (CompositePropertySource) aPropSource;
cps.getPropertySources().forEach( ps -> addAll( result, getAllProperties( ps ) ) );
return result;
}
if ( aPropSource instanceof EnumerablePropertySource<?> )
{
EnumerablePropertySource<?> ps = (EnumerablePropertySource<?>) aPropSource;
Arrays.asList( ps.getPropertyNames() ).forEach( key -> result.put( key, ps.getProperty( key ) ) );
return result;
}
// note: Most descendants of PropertySource are EnumerablePropertySource. There are some
// few others like JndiPropertySource or StubPropertySource
myLog.debug( "Given PropertySource is instanceof " + aPropSource.getClass().getName()
+ " and cannot be iterated" );
return result;
}
private static void addAll( Map<String, Object> aBase, Map<String, Object> aToBeAdded )
{
for (Entry<String, Object> entry : aToBeAdded.entrySet())
{
if ( aBase.containsKey( entry.getKey() ) )
{
continue;
}
aBase.put( entry.getKey(), entry.getValue() );
}
}
Note that the starting point is the ConfigurableEnvironment which is able to return the embedded PropertySources (the ConfigurableEnvironment is a direct descendant of Environment). You can autowire it by:
#Autowired
private ConfigurableEnvironment myEnv;
If you not using very special kinds of property sources (like JndiPropertySource, which is usually not used in spring autoconfiguration) you can retrieve all properties held in the environment.
The implementation relies on the iteration order which spring itself provides and takes the first found property, all later found properties with the same name are discarded. This should ensure the same behaviour as if the environment were asked directly for a property (returning the first found one).
Note also that the returned properties are not yet resolved if they contain aliases with the ${...} operator. If you want to have a particular key resolved you have to ask the Environment directly again:
myEnv.getProperty( key );
The original question hinted that it would be nice to be able to filter all the properties based on a prefix. I have just confirmed that this works as of Spring Boot 2.1.1.RELEASE, for Properties or Map<String,String>. I'm sure it's worked for while now. Interestingly, it does not work without the prefix = qualification, i.e. I do not know how to get the entire environment loaded into a map. As I said, this might actually be what OP wanted to begin with. The prefix and the following '.' will be stripped off, which might or might not be what one wants:
#ConfigurationProperties(prefix = "abc")
#Bean
public Properties getAsProperties() {
return new Properties();
}
#Bean
public MyService createService() {
Properties properties = getAsProperties();
return new MyService(properties);
}
Postscript: It is indeed possible, and shamefully easy, to get the entire environment. I don't know how this escaped me:
#ConfigurationProperties
#Bean
public Properties getProperties() {
return new Properties();
}
As this Spring's Jira ticket, it is an intentional design. But the following code works for me.
public static Map<String, Object> getAllKnownProperties(Environment env) {
Map<String, Object> rtn = new HashMap<>();
if (env instanceof ConfigurableEnvironment) {
for (PropertySource<?> propertySource : ((ConfigurableEnvironment) env).getPropertySources()) {
if (propertySource instanceof EnumerablePropertySource) {
for (String key : ((EnumerablePropertySource) propertySource).getPropertyNames()) {
rtn.put(key, propertySource.getProperty(key));
}
}
}
}
return rtn;
}
Spring won't allow to decouple via java.util.Properties from Spring Environment.
But Properties.load() still works in a Spring boot application:
Properties p = new Properties();
try (InputStream is = getClass().getResourceAsStream("/my.properties")) {
p.load(is);
}
The other answers have pointed out the solution for the majority of cases involving PropertySources, but none have mentioned that certain property sources are unable to be casted into useful types.
One such example is the property source for command line arguments. The class that is used is SimpleCommandLinePropertySource. This private class is returned by a public method, thus making it extremely tricky to access the data inside the object. I had to use reflection in order to read the data and eventually replace the property source.
If anyone out there has a better solution, I would really like to see it; however, this is the only hack I have gotten to work.
Working with Spring Boot 2, I needed to do something similar. Most of the answers above work fine, just beware that at various phases in the app lifecycles the results will be different.
For example, after a ApplicationEnvironmentPreparedEvent any properties inside application.properties are not present. However, after a ApplicationPreparedEvent event they are.
For Spring Boot, the accepted answer will overwrite duplicate properties with lower priority ones. This solution will collect the properties into a SortedMap and take only the highest priority duplicate properties.
final SortedMap<String, String> sortedMap = new TreeMap<>();
for (final PropertySource<?> propertySource : env.getPropertySources()) {
if (!(propertySource instanceof EnumerablePropertySource))
continue;
for (final String name : ((EnumerablePropertySource<?>) propertySource).getPropertyNames())
sortedMap.computeIfAbsent(name, propertySource::getProperty);
}
I though I'd add one more way. In my case I supply this to com.hazelcast.config.XmlConfigBuilder which only needs java.util.Properties to resolve some properties inside the Hazelcast XML configuration file, i.e. it only calls getProperty(String) method. So, this allowed me to do what I needed:
#RequiredArgsConstructor
public class SpringReadOnlyProperties extends Properties {
private final org.springframework.core.env.Environment delegate;
#Override
public String getProperty(String key) {
return delegate.getProperty(key);
}
#Override
public String getProperty(String key, String defaultValue) {
return delegate.getProperty(key, defaultValue);
}
#Override
public synchronized String toString() {
return getClass().getName() + "{" + delegate + "}";
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
if (!super.equals(o)) return false;
SpringReadOnlyProperties that = (SpringReadOnlyProperties) o;
return delegate.equals(that.delegate);
}
#Override
public int hashCode() {
return Objects.hash(super.hashCode(), delegate);
}
private void throwException() {
throw new RuntimeException("This method is not supported");
}
//all methods below throw the exception
* override all methods *
}
P.S. I ended up not using this specifically for Hazelcast because it only resolves properties for XML file but not at runtime. Since I also use Spring, I decided to go with a custom org.springframework.cache.interceptor.AbstractCacheResolver#getCacheNames. This resolves properties for both situations, at least if you use properties in cache names.
To get ONLY properties, defined in my hibernate.properteies file:
#PropertySource(SomeClass.HIBERNATE_PROPERTIES)
public class SomeClass {
public static final String HIBERNATE_PROPERTIES = "hibernate.properties";
#Autowired
private Environment env;
public void someMethod() {
final Properties hibProps = asProperties(HIBERNATE_PROPERTIES);
}
private Properties asProperties(String fileName) {
return StreamSupport.stream(
((AbstractEnvironment) env).getPropertySources().spliterator(), false)
.filter(ps -> ps instanceof ResourcePropertySource)
.map(ps -> (ResourcePropertySource) ps)
.filter(rps -> rps.getName().contains(fileName))
.collect(
Properties::new,
(props, rps) -> props.putAll(rps.getSource()),
Properties::putAll);
}
}
A little helper to analyze the sources of a property, which sometimes drive me crazy . I used this discussion to write SpringConfigurableEnvironment.java on github.
It could be used in a test:
#SpringBootTest
public class SpringConfigurableEnvironmentTest {
#Autowired
private ConfigurableEnvironment springEnv;
#Test
public void testProperties() {
SpringConfigurableEnvironment properties = new SpringConfigurableEnvironment(springEnv);
SpringConfigurableEnvironment.PropertyInfo info = properties.get("profile.env");
assertEquals("default", properties.get(info.getValue());
assertEquals(
"Config resource 'class path resource [application.properties]' via location 'optional:classpath:/'",
info.getSourceList.get(0));
}
}
All answers above have pretty much covers everything, but be aware of overridden values from environment variables. They may have different key values.
For example, if a user override my.property[1].value using environment variable MY_PROPERTY[1]_VALUE, iterating through EnumerablePropertySources.getPropertyNames() would give you both my.property[1].value and MY_PROPERTY[1]_VALUE key values.
What even worse is that if my.property[1].value is not defined in applications.conf (or applications.yml), a MY_PROPERTY[1]_VALUE in environment variables would not give you my.property[1].value but only MY_PROPERTY[1]_VALUE key value from EnumerablePropertySources.getPropertyNames().
So it is developers' job to cover the those properties from environment variables. Unfortunately, there is no one-on-one mapping between environment variables schema vs the normal schema, see the source code of SystemEnvironmentPropertySource. For example, MY_PROPERTY[1]_VALUE could be either my.property[1].value or my-property[1].value
I am using Guava to cache hot data. When the data does not exist in the cache, I have to get it from database:
public final static LoadingCache<ObjectId, User> UID2UCache = CacheBuilder.newBuilder()
//.maximumSize(2000)
.weakKeys()
.weakValues()
.expireAfterAccess(10, TimeUnit.MINUTES)
.build(
new CacheLoader<ObjectId, User>() {
#Override
public User load(ObjectId k) throws Exception {
User u = DataLoader.datastore.find(User.class).field("_id").equal(k).get();
return u;
}
});
My problem is when the data does not exists in database, I want it to return null and to not do any caching. But Guava saves null with the key in the cache and throws an exception when I get it:
com.google.common.cache.CacheLoader$InvalidCacheLoadException:
CacheLoader returned null for key shisoft.
How do we avoid caching null values?
Just throw some Exception if user is not found and catch it in client code while using get(key) method.
new CacheLoader<ObjectId, User>() {
#Override
public User load(ObjectId k) throws Exception {
User u = DataLoader.datastore.find(User.class).field("_id").equal(k).get();
if (u != null) {
return u;
} else {
throw new UserNotFoundException();
}
}
}
From CacheLoader.load(K) Javadoc:
Returns:
the value associated with key; must not be null
Throws:
Exception - if unable to load the result
Answering your doubts about caching null values:
Returns the value associated with key in this cache, first loading
that value if necessary. No observable state associated with this
cache is modified until loading completes.
(from LoadingCache.get(K) Javadoc)
If you throw an exception, load is not considered as complete, so no new value is cached.
EDIT:
Note that in Caffeine, which is sort of Guava cache 2.0 and "provides an in-memory cache using a Google Guava inspired API" you can return null from load method:
Returns:
the value associated with key or null if not found
If you may consider migrating, your data loader could freely return when user is not found.
Simple solution: use com.google.common.base.Optional<User> instead of User as value.
public final static LoadingCache<ObjectId, Optional<User>> UID2UCache = CacheBuilder.newBuilder()
...
.build(
new CacheLoader<ObjectId, Optional<User>>() {
#Override
public Optional<User> load(ObjectId k) throws Exception {
return Optional.fromNullable(DataLoader.datastore.find(User.class).field("_id").equal(k).get());
}
});
EDIT: I think #Xaerxess' answer is better.
Faced the same issue, cause missing values in the source was part of the normal workflow. Haven't found anything better than to write some code myself using getIfPresent, get and put methods. See the method below, where local is Cache<Object, Object>:
private <K, V> V getFromLocalCache(K key, Supplier<V> fallback) {
#SuppressWarnings("unchecked")
V s = (V) local.getIfPresent(key);
if (s != null) {
return s;
} else {
V value = fallback.get();
if (value != null) {
local.put(key, value);
}
return value;
}
}
When you want to cache some NULL values, you could use other staff which namely behave as NULL.
And before give the solution, I would suggest you not to expose LoadingCache to outside. Instead, you should use method to restrict the scope of Cache.
For example, you could use LoadingCache<ObjectId, List<User>> as return type. And then, you could return empty list when you could'n retrieve values from database. You could use -1 as Integer or Long NULL value, you could use "" as String NULL value, and so on. After this, you should provide a method to handler the NULL value.
when(value equals NULL(-1|"")){
return null;
}
I use the getIfPresent
#Test
public void cache() throws Exception {
System.out.println("3-------" + totalCache.get("k2"));
System.out.println("4-------" + totalCache.getIfPresent("k3"));
}
private LoadingCache<String, Date> totalCache = CacheBuilder
.newBuilder()
.maximumSize(500)
.refreshAfterWrite(6, TimeUnit.HOURS)
.build(new CacheLoader<String, Date>() {
#Override
#ParametersAreNonnullByDefault
public Date load(String key) {
Map<String, Date> map = ImmutableMap.of("k1", new Date(), "k2", new Date());
return map.get(key);
}
});