nHibernate second-level cache with memcached and random invalid casts - asp.net-mvc-3

Question

I have an MVC3 application using nHibernate and memcached as the second level cache provider. We've been intermittently (but much more frequently lately) getting weird casting issues. It happens randomly and invalidating the memcached cache will fix the problem for while.
It only happens in our Production environment because we don't run memcached in our other environments. However I run memcached locally and have tried to get this to occur locally with no luck.
We are using memcached 1.2.6 on Windows. Here's the stack trace. I know it's not going to be enough information to determine anything but if anyone has any ideas about how I could debug this that would be appreciated. I'm trying to get remote debugging on our Production machine but it's a busy time of year and risky.
Unable to cast object of type 'System.Int32' to type 'System.String'.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.InvalidCastException: Unable to cast object of type 'System.Int32' to type 'System.String'.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[InvalidCastException: Unable to cast object of type 'System.Int32' to type 'System.String'.]
(Object , Object[] , SetterCallback ) +4270
NHibernate.Tuple.Entity.PocoEntityTuplizer.SetPropertyValuesWithOptimizer(Object entity, Object[] values) +80
[PropertyAccessException: Invalid Cast (check your mapping for property type mismatches); setter of MyApplication.Business.Data.Program]
NHibernate.Tuple.Entity.PocoEntityTuplizer.SetPropertyValuesWithOptimizer(Object entity, Object[] values) +207
NHibernate.Tuple.Entity.PocoEntityTuplizer.SetPropertyValues(Object entity, Object[] values) +97
NHibernate.Cache.Entry.CacheEntry.Assemble(Object[] values, Object result, Object id, IEntityPersister persister, IInterceptor interceptor, ISessionImplementor session) +306
NHibernate.Cache.Entry.CacheEntry.Assemble(Object instance, Object id, IEntityPersister persister, IInterceptor interceptor, ISessionImplementor session) +147
NHibernate.Event.Default.DefaultLoadEventListener.AssembleCacheEntry(CacheEntry entry, Object id, IEntityPersister persister, LoadEvent event) +434
NHibernate.Event.Default.DefaultLoadEventListener.LoadFromSecondLevelCache(LoadEvent event, IEntityPersister persister, LoadType options) +800
NHibernate.Event.Default.DefaultLoadEventListener.DoLoad(LoadEvent event, IEntityPersister persister, EntityKey keyToLoad, LoadType options) +560
NHibernate.Event.Default.DefaultLoadEventListener.Load(LoadEvent event, IEntityPersister persister, EntityKey keyToLoad, LoadType options) +229
NHibernate.Event.Default.DefaultLoadEventListener.ProxyOrLoad(LoadEvent event, IEntityPersister persister, EntityKey keyToLoad, LoadType options) +438
NHibernate.Event.Default.DefaultLoadEventListener.OnLoad(LoadEvent event, LoadType loadType) +943
NHibernate.Impl.SessionImpl.FireLoad(LoadEvent event, LoadType loadType) +99
NHibernate.Impl.SessionImpl.Get(String entityName, Object id) +117
NHibernate.Impl.SessionImpl.Get(Object id) +70
MyApplication.Business.Repositories.Repository`1.Get(Object id) +148

Answer 1

We've been encountering the same issue and we have pinpointed the problem to invalid cache entries in the second level cache. In our case the problem arises when we add or remove columns/properties from tables/entities that are cached in the second level cache and deploy the change. The code expects five columns (for argument's sake) whereas the cache stores six columns.
One strategy that we employed in our repository is to catch the exception and invalidate the cache. We then retry the get.
public override T Get(int id)
{
try
{
return base.Get(id);
}
catch (InvalidCastException)
{
_sessionFactory.EvictEntity(typeof (T).Name);
return Get(id);
}
}
Hope this helps!

Answer 2

Most probably this is caused by different dataset returned by the queries on the production environment and your local. The object mapper is getting different data than what its expecting. When trying to convert the data then you are getting these exceptions.