NuGet Gallery and EntityFramework throws Failed to open provider error - asp.net-mvc-3

Question

We have a weird situation going on here. We have setup a local installation of NuGet gallery (the same code that powers http://Nuget.org). The IIS and the DB are on the same box. The IIS App pool has been configured to run under a domain user who is the local admin on the box as well and has the "right to logon as a service" rights. Now when someone (after succesfully logging in) tries and upload a package, the upload package business takes ages and eventually throws up the "The underlying provider failed to open" error
[TimeoutException: Transaction Timeout]
[TransactionException: The operation is not valid for the state of the transaction.]
System.Transactions.TransactionState.EnlistPromotableSinglePhase(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification, Transaction atomicTransaction) +53
System.Transactions.Transaction.EnlistPromotableSinglePhase(IPromotableSinglePhaseNotification promotableSinglePhaseNotification) +241
System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx) +273
System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction) +150
System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject) +2647
System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection) +89
System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +6372062
System.Data.SqlClient.SqlConnection.Open() +300
System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean& closeStoreConnectionOnFailure) +67
[EntityException: The underlying provider failed on Open.]
System.Data.EntityClient.EntityConnection.OpenStoreConnectionIf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean& closeStoreConnectionOnFailure) [phone]
System.Data.EntityClient.EntityConnection.Open() +142
System.Data.Objects.ObjectContext.EnsureConnection() +97
System.Data.Objects.ObjectContext.ExecuteStoreQueryInternal(String commandText, String entitySetName, MergeOption mergeOption, Object[] parameters) +109
System.Data.Objects.ObjectContext.ExecuteStoreQuery(String commandText, Object[] parameters) +87
System.Data.Entity.Internal.InternalContext.ExecuteSqlQuery(String sql, Object[] parameters) +118
System.Data.Entity.Internal.InternalContext.ExecuteSqlQueryAsIEnumerable(String sql, Object[] parameters) +85
System.Data.Entity.Internal.InternalContext.ExecuteSqlQuery(Type elementType, String sql, Object[] parameters) +241
System.Data.Entity.Internal.InternalSqlNonSetQuery.GetEnumerator() +34
System.Data.Entity.Internal.InternalSqlQuery`1.GetEnumerator() +28
System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) +382
System.Linq.Enumerable.ToList(IEnumerable`1 source) +80
NuGetGallery.LuceneIndexingService.GetPackages(DbContext context, Nullable`1 dateTime) in C:\Source\DotNetDevEng.CitiNuGet\Gallery\DEV\Website\Infrastructure\Lucene\LuceneIndexingService.cs:55
NuGetGallery.LuceneIndexingService.UpdateIndex() in C:\Source\DotNetDevEng.CitiNuGet\Gallery\DEV\Website\Infrastructure\Lucene\LuceneIndexingService.cs:32
NuGetGallery.PackageService.CreatePackage(IPackage nugetPackage, User currentUser) in C:\Source\DotNetDevEng.CitiNuGet\Gallery\DEV\Website\Services\PackageService.cs:61
NuGetGallery.PackagesController.VerifyPackage(Nullable`1 listed) in C:\Source\DotNetDevEng.CitiNuGet\Gallery\DEV\Website\Controllers\PackagesController.cs:503
lambda_method(Closure , ControllerBase , Object[] ) +121
System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters) +248
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +39
System.Web.Mvc.<>c__DisplayClass15.<InvokeActionMethodWithFilters>b__12() +125
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +640
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodFilter(IActionFilter filter, ActionExecutingContext preContext, Func`1 continuation) +640
System.Web.Mvc.ControllerActionInvoker.InvokeActionMethodWithFilters(ControllerContext controllerContext, IList`1 filters, ActionDescriptor actionDescriptor, IDictionary`2 parameters) +312
System.Web.Mvc.ControllerActionInvoker.InvokeAction(ControllerContext controllerContext, String actionName) +691
System.Web.Mvc.Controller.ExecuteCore() +162
System.Web.Mvc.ControllerBase.Execute(RequestContext requestContext) +305
System.Web.Mvc.<>c__DisplayClassb.<BeginProcessRequest>b__5() +62
System.Web.Mvc.Async.<>c__DisplayClass1.<MakeVoidDelegate>b__0() +20
System.Web.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +469
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +375
Interestingly if I switch IIS App Pool to run under the built in "LocalSystem" account all these problems goes away. The domain user we want to use has been temporarily setup as the database owner for NuGetGallery database but it isn't helping.
Can anyone suggest what setting/configuration we might be missing here?

Answer 1

This was due to ambient transactions going on within the code. The code relied on TransactionScope classes to manage various transactions which were getting in a jam when they got nested. Things turned particularly bad if someone tried uploading a package which was more than around 10MB in size. Once we tweaked the MSDTC on the boxes the application started behaving slightly properly. However it was still intermittently getting deadlocked within its own transactions so for now we have decided to remove transactions from the package upload bit.
For anyone else who gets into this problem I would recommend either of these two
Comment out any reference to TransactionScope in
Services\PackageService.cs (method CreatePackage, line 48)
using (var tx = new TransactionScope())
{
using (var stream = nugetPackage.GetStream())
{
UpdateIsLatest(packageRegistration);
packageRegistrationRepo.CommitChanges();
packageFileSvc.SavePackageFile(package, stream);
tx.Complete();
}
}
And in Controller\PackageController.cs method VerifyPackage(bool? Listed) line 501
using (var tx = new TransactionScope())
{
package = packageSvc.CreatePackage(nugetPackage, currentUser);
packageSvc.PublishPackage(package.PackageRegistration.Id, package.Version);
if (listed.HasValue && listed.Value == false)
packageSvc.MarkPackageUnlisted(package);
uploadFileSvc.DeleteUploadFile(currentUser.Key);
tx.Complete();
}
2 Uncomment those lines and follow the approach mentioned here http://blogs.msdn.com/b/dbrowne/archive/2010/05/21/using-new-transactionscope-considered-harmful.aspx
Basically create a transactionscope which can work well with SQL Server. Create a new class in the InfraStructureFolder, call it TranscationUtils and copy the code in there. Then the using blocks would look like
using (var tx = new TransactionUtils.CreateTransactionScope()){
//cod here
tx.Complete();
}
However this second approach can result in another server error which again relates to the default time outs you can specify for TransactionScope. We have personally decided to go with approach 1 and comment out all TransactionScope references for now.