we are using Appfabric cache in our project, and we ran into 2 major problems.
First - we are using named caches (no explicitly created regions). One of them, created as Expirable=false, Eviction=none, TTL=525600 is used for objects that should be always available (populated at application start, via the Put method). But from time to time (i couldnt identify exact timespan, nor connection to certain actions in application) all object in this cache suddenly expires - i can see this from performance counters - object count for this cache goes to 0, total expired objects counter increases of the amount of objects in this cache at the same time. Am i missing some other settings ? I tried both inserting them via Put() without timespan, and Put with timespan "a year". Still expires after several minutes...
The second problem - when I tried to solve first problem, i decided to use ETW trace logging feature to see in log, what is happening. I have created tracelog via logman and started it, waited for cache to expire, stopped the log, and have used tracerpt to create dumpfile from etl. Everything ok so far. But this dumpfile is useless, because there are no readable data, only 4400690073007400720....... After some quick research, i figured out, that I need to supply an PDB or TMF file to tracerpt, so it can "decode" binaryeventdata to readable eventdata. Is it possible to get some of these for appfabric cache ? Or there is some other way to use ETW with appfabric to get some usefull readable log ?
I found out what the problem is when your cache is expiring nearly instantly.
If your memory is low the cache gets cleared.
Check in the eventvwr -> Applicatin and Service -> Microsoft -> Windows - Application Server System Services and select Operational.
Look for warnings like:
Service available memory low - Cache private bytes percent {2} Cache working set percent {1} Cache data size percent {0} Available memory percent {21} CLR Generation2 count {2013} Released memory percent {0}.
There is an explanation in here to convert log file to cvs: http://msdn.microsoft.com/en-us/library/ff921010.aspx
But I can not even use tracelog tool.
Related
I'm using Sitecore instance 9.1, Solr 7.2.1, and SXA 1.8.
I have deployed the environment on Azure and while monitoring incoming requests (to CD instance), I've noticed slowness in loading some pages at specific times.
I've explored App Insights and found an unexplainable behavior the request is taking 28.7 seconds while the breakdown of it shows executions of milli-seconds .. How is that possible? and How to explain what's happening during extra 28 seconds on the app service ??
I've checked the profiler and it shows that the thread is taking only 1042.48 ms .. How is that possible ?
This is an intermittent issue happens during the day .. regular requests are being served within 3 to 4 seconds.
I noticed that Azure often shows a profile trace for a "similar", but completely different request when clicking from the End-to-end transaction view. You can check this by comparing the timestamp and URL of the profile trace and the transaction you clicked from.
For example, I see a transaction logged at 8:58:39 PM, 2021-09-25 with 9.1 s response time:
However, when I click the profile trace icon, Azure takes me to a trace that was captured 10 minutes earlier, at 08:49:20 PM, 2021-09-25 and took only 121.64 ms:
So, if the issue you experience is intermittent and you cannot replicate it easily, try looking at the profile traces with the Slowest wall clock time by going to Application Insights → Performance → Drill into profile traces:
This will show you the worst-performing requests captured by the profiler at the top of the list:
In order to figure out why it is slow, you’ll need to understand what happens internally, f.e:
How the wall clock time is spent while processing your request?
Are there any locks internally?
The source of that data is dynamic profiling, Azure can do that on demand.
The IIS stats report would show you slowest requests, so you could look into Thread Time distribution to see where those 28 seconds are spent:
In Sitecore the when the application start the Initial prefetch configuration allows to pre-populate prefetch caches. Pre-heated prefetch caches help to reduce the processing time of incoming requests. The initial prefetch configuration of caches are taking time to load on initial stage.
Sitecore XP instance takes too long to load. This is caused by a performance issue in the CatalogRepository.GetCatalogItems method. It will be fixed in upcoming updates
see Site core knowledge base
In Sitecore XP 9.0 the initial prefetch configuration was revised. The prefetch cache for the core database was configured to include items that are used to render the Sitecore Client interface.
The Sitecore Client interface is not used on Content Delivery instances. Disabling initial prefetch configuration for the Core database helps in avoiding excessive resource consumption on the SQL Server hosting the Core database.
Change the configuration of the Core database in the \App_Config\Sitecore.config file:
Refer site core knowledge base
I am trying to implement temporary ip blocking for my ASP.NET Core application and everything I have read on how to do this uses caching.
I have followed this article to learn about in-memory caching in asp.net core:
https://learn.microsoft.com/en-us/aspnet/core/performance/caching/memory
The code is running okay, but the cache is emptied immediately after an action is completed.
If a put a break-point on this line:
_cache.Set(CacheKeys.Entry, cacheEntry, cacheEntryOptions);
I can see that _cache indeed has the proper value. However, that value is gone the next time I reload the page. Regardless of how long I set the expiration time to or even if I set
CacheItemPriority.NeverRemove
in the cacheEntryOptions.
I have also tried registering a callback for the eviction of the item from the cache to see if it was being evicted immediately due to memory pressure. However, the callback is never called.
Is there some server options I need to enable to allow caching?
I met the similar issue today. And found that the cache only save the reference of the value. So if you save a List in to the cache and Clear the List later. The cached List will also be cleared, but key is still there.
Source code is here:https://source.dot.net/#Microsoft.Extensions.Caching.Abstractions/MemoryCacheExtensions.cs,bc5684b7eae179d4
With the Azure Cache service, I am trying to find details of the following aspects of the service:
What is the maximum length of time something can be kept in the cache? I presume it would be until the cache service is restarted;
Is there a way to detect that the cache service has been restarted?
My intention is to use Azure Cache to store datasets that are frequently being accessed, and that would be updated / added to over time as data that is incoming into my system is processed.
How would I know / be notified that the cache has restarted (I guess apart from seeing if it is empty) so I could kick off a process to repopulate it?
I think you'll find answer to your questions here: http://msdn.microsoft.com/en-us/library/dn386128.aspx
Alright, information on High Availability of the cache service is here: http://msdn.microsoft.com/en-us/library/dn386134.aspx
I am using Windows Azure SDK 2.2 and have created an Azure cloud service that uses an in-role cache.
I have 2 instances of the service running under normal conditions.
When the services scales (up to 3 instances, or back down to 2 instances), I get lots of DataCacheExceptions. These are often accompanied by Azure db connection failures from the process going in inside the cache. (If I don't find the entry I want in the cache, I get it from the db and put it into the cache. All standard stuff.)
I have implemented retry processes on the cache gets and puts, and use the ReliableSqlConnection object with a retry process for db connection using the Transient Fault Handling application block.
The retry process uses a fixed interval retrying every second for 5 tries.
The failures are typically;
Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode:SubStatus:There is a temporary failure. Please retry later
Any idea why the scaling might cause these exceptions?
Should I try a less aggressive retry policy?
Any help appreciated.
I have also noticed that I am getting a high percentage (> 70%) cache miss rate and when the system is struggling, there is high cpu utilisation (> 80%).
Well, I haven't been able to find out any reason for the errors I am seeing, but I have 'fixed' the problem, sort of!
When looking at the last few days processing stats, it is clear the high cpu usage corresponds with the cloud service having 'problems'. I have changed the service to use two medium instances instead of two small instances.
This seems to have solved the problem, and the service has been running quite happily, low cpu usage, low memory usage, no exceptions.
So, whilst still not discovering what the source of the problems were, I seem to have overcome them by providing a bigger environment for the service to run in.
--Late news!!! I noticed this morning that from about 06:30, the cpu usage started to climb, along with the time taken for the service to process as it should. Errors started appearing and I had to restart the service at 10:30 to get things back to 'normal'. Also, when restarting the service, the OnRoleRun process threw loads of DataCacheExceptions before it started running again, 45 minutes later.
Now all seems well again, and I will monitor for the next hours/days...
There seems to be no explanation for this, remote desktop to the instances show no exceptions in the event log, other logging is not showing application problems, so I am still stumped.
I have a default cache that is fairly small and static. It contains just string keys and a string objects.
Since I won't be using anywhere near the allowed amount of memory, I'd like to just preload all of the objects into the cache on startup and have them never expire. I added a log message on start indicating that the cache was loaded.
Right now the project is still in development so the cache isn't being hit often (other than by web spiders/crawlers/scripts). The problem I'm seeing is that every hour to few hours, I'm seeing the log message that my cache was loaded. I'd expect it to load once and then not reload until I force it to.
Is there any way to keep the cache "alive" so that it doesn't have to frequently reload? Is it like an IIS worker process that dies out after some amount of inactivity?
FYI I have the cache configured for Expiry Policy: Never, Time: 0min, Eviction: Disabled. Also the way I check if the cache is still alive is that on load I add a special object to the cache. Then I check to see if that object exists and if it doesn't I assume the cache needs to be reloaded.
For anyone else who stumbles across this I ended up creating a scheduled task that hit the cache every 5 minutes. Since then, I haven't seemed to have any issues with it reloading. Not sure if this is the best answer, but it worked for me.