Automapper first IQueryable.ProjectTo projection is too slow - performance

I have Dto classes:
public class PackageDto
{
public MemberDto Receiver { get; set; }
public MemberDto Sender { get; set; }
public DocflowDto Docflow { get; set; }
public DocflowTypeOption DocflowTypeId { get; set; }
public string Code { get; set; }
public Guid? DocGuid { get; set; }
}
public class LightPackageDto
{
public DocflowDto Docflow { get; set; }
public DocflowTypeOption DocflowTypeId { get; set; }
public string Code { get; set; }
public Guid? DocGuid { get; set; }
}
public class MemberDto
{
public string Prop1 { get; set; }
public EntityTypeOption Prop2 { get; set; }
public string Prop3 { get; set; }
public string Prop4 { get; set; }
public string Prop5 { get; set; }
public PersonDto Director { get; set; }
public string CompanyShortName { get; set; }
public string CompanyFullName { get; set; }
public string CompanyPrefix { get; set; }
}
DocflowDto has 3 simple type properties and is projected by flattening.
PersonDto has 3 simple type properties and is projected by flattening too
Mapping rules:
CreateMap<Person, PersonDto>();
CreateMap<Member, MemberDto>()
.ForMember(dest => dest.Prop1, opt => opt.MapFrom(src => src.Employee.Prop1))
.ForMember(dest => dest.Prop2, opt => opt.MapFrom(src => src.Prop2))
.ForMember(dest => dest.Prop3, opt => opt.MapFrom(src => src.Employee.Prop3))
.ForMember(dest => dest.Prop4, opt => opt.MapFrom(src => src.Employee.Prop4))
.ForMember(dest => dest.Prop5, opt => opt.MapFrom(src => src.Employee.Prop5))
.ForMember(dest => dest.Director, opt => opt.MapFrom(src => src.Employee.Director))
.ForMember(dest => dest.CompanyShortName, opt => opt.MapFrom(src => src.Company.CompanyShortName))
.ForMember(dest => dest.CompanyFullName, opt => opt.MapFrom(src => src.Company.CompanyFullName))
.ForMember(dest => dest.CompanyPrefix, opt => opt.MapFrom(src => src.Company.Prefix))
;
CreateMap<Data.Models.Docflow.Package, PackageDto>()
.ForMember(dest => dest.Sender, opt => opt.MapFrom(src => src.Sender))
.ForMember(dest => dest.Receiver, opt => opt.MapFrom(src => src.Receiver))
.ForMember(dest => dest.Docflow, opt => opt.MapFrom(src => src.Docflow))
.ForMember(dest => dest.DocGuid, opt => opt.MapFrom(src => src.Files.FirstOrDefault().DocGuid))
;
CreateMap<Data.Models.Docflow.Package, LightPackageDto>()
.ForMember(dest => dest.Docflow, opt => opt.MapFrom(src => src.Docflow))
.ForMember(dest => dest.DocGuid, opt => opt.MapFrom(src => src.Files.FirstOrDefault().DocGuid));
Then tests:
//fast
var lightEntity = await _db.Packages.Where(p => p.PackageGuid == packageGuid).ProjectTo<LightPackageDto>().FirstOrDefaultAsync();
//fast (_db.GetPackage make query with all necessary includes)
var entity = await _db.GetPackage(packageGuid);
var entityDto = AutoMapper.Mapper.Map<Package, PackageDto>(entity);
//first run too slow
var result = await _db.Packages.AsNoTracking().Where(p => p.PackageGuid == packageGuid).ProjectTo<PackageDto>().FirstOrDefaultAsync();
1) So if I project to LightPackageDto (without Member projections) is all right.
2) If I return entity manually from EF with includes and then I make mappings for "in memory" entity is all right too
3) If I want Automapper makes "Select" query applying ProjectTo I allways get freeze about 2-3 sec. But freeze appears only single first time. Why?
And additional question. I see in result sql select statement all the columns of all the concerning tables. I supposed to see only columns needed for destination projection. Why are all there?

It is just some stupid bug. It is not caching or anything. Because touching the intended projection and canceling the task is enough to "clean the path" (solution 1). Or (sometimes) converting query to list before getting first element also works (solution 2).
Solution 1:
Works every time for me. (Note that the time (1000 ms in this case) starts counting when CancelAfter method is hit. So debugging step-by-step jumps to catch block almost every time.)
CancellationTokenSource cancelationTokenSource = new CancellationTokenSource();
cancelationTokenSource.CancelAfter(1000);
PackageDto result = null;
try
{
result = await _db.Packages.AsNoTracking()
.Where(p => p.PackageGuid == packageGuid).ProjectTo<PackageDto>()
.FirstOrDefaultAsync(cancelationTokenSource.Token);//cancel this, when taking longer than 1s
}
catch (OperationCanceledException)
{ //do absolutely the same thing, but now it will be fast...
result = await _db.Packages.AsNoTracking()
.Where(p => p.PackageGuid == packageGuid).ProjectTo<PackageDto>()
.FirstOrDefaultAsync();
}
Solution 2:
!!! Doesn't work consistently. Need some more investigation
PackageDto result = await _db.Packages.AsNoTracking()
.Where(p => p.PackageGuid == packageGuid).ProjectTo<PackageDto>()
.ToList().FirstOrDefault();//converting to list for some reason helps here..
Yeah. Not the most beautiful solution, but better than waiting 25 seconds...
(btw .net6, Automapper 10.1.1, AutoMapper.Collection.EntityFrameworkCore 7.1.3)

Related

How can I execute ElasticSearch query on multiple indices with nested mapping

I have two indices with the following configuration with mappings
var settings = new ConnectionSettings(new Uri("http://localhost:9200/"));
settings
.DefaultMappingFor<ManagementIndex>(m => m
.IndexName("management")
)
.DefaultMappingFor<PropertyIndex>(m => m
.IndexName("apartmentproperty")
);
var client = new ElasticClient(settings);
1) Properties mapping
client.Indices.Create("property", i => i
.Settings(s => s
.NumberOfShards(2)
.NumberOfReplicas(0)
)
.Map<PropertyIndex>(map => map
.AutoMap()
.Properties(p => p
.Nested<PropertyData>(n => n
.Name(c => c.property)
.AutoMap()
.Properties(pp => pp
.Text(c => c
.Name(np => np.city)
.Analyzer("standard")
)
.Text(c => c
.Name(np => np.market)
.Fields(ff => ff
.Text(tt => tt
.Name(np => np.market)
.Analyzer("standard")
)
.Keyword(k => k
.Name("keyword")
.IgnoreAbove(256)
)
)
).Text(c => c
.Name(np => np.name)
.Analyzer("standard")
)
)
)
)
)
);
and
2) Owner
if (client.Indices.Exists("owner").Exists)
client.Indices.Delete("owner");
client.Indices.Create("owner", i => i
.Settings(s => s
.NumberOfShards(2)
.NumberOfReplicas(0)
)
.Map<OwnerIndex>(map => map
.AutoMap()
.Properties(p => p
.Nested<OwnerProp>(n => n
.Name(c => c.owner)
.AutoMap()
.Properties(pp => pp
.Text(c => c
.Name(np => np.market)
.Fields(ff => ff
.Text(tt => tt
.Name(np => np.market)
.Analyzer("standard")
)
.Keyword(k => k
.Name("keyword")
.IgnoreAbove(256)
)
)
).Text(c => c
.Name(np => np.name)
.Analyzer("standard")
)
)
)
)
)
);
with the following POCO definitions
public class PropertyData
{
public string name { get; set; }
public string city { get; set; }
public string market { get; set; }
}
public class PropertyIndex
{
public PropertyData property { get; set; }
}
public class OwnerProp
{
public string name { get; set; }
public string market { get; set; }
}
public class OwnerIndex
{
public OwnerProp owner { get; set; }
}
Trying to do a search through the two indices like so
public async Task<object> SearchPropertiesAsync(string searchQuery, List<string> description, int limit = 25, int skip = 1)
{
var propertyfilters = new List<Func<QueryContainerDescriptor<object>, QueryContainer>>();
var ownerFilters = new List<Func<QueryContainerDescriptor<object>, QueryContainer>>();
if (description.Any())
{
propertyfilters.Add(fq => fq.Terms(t => t.Field("property.market.keyword").Terms(description)));
ownerFilters.Add(fq => fq.Terms(t => t.Field("owner.market.keyword").Terms(description)));
}
var searchResponse = await _elasticClient.SearchAsync<object>(s => s
.Index(Indices.Index(typeof(PropertyIndex)).And(typeof(OwnerIndex)))
.Query(q => (q
.Nested(n => n
.Path(Infer.Field<PropertyIndex>(ff => ff.property))
.Query(nq => nq
.MultiMatch(m => m
.Fields(f => f
.Field(Infer.Field<PropertyIndex>(ff => ff.property.city))
.Field(Infer.Field<PropertyIndex>(ff => ff.property.market))
.Field(Infer.Field<PropertyIndex>(ff => ff.property.name))
)
.Operator(Operator.Or)
.Query(searchQuery)
.Fuzziness(Fuzziness.Auto)
) && +q.Bool(bq => bq.Filter(propertyfilters))
))
) || (q
.Nested(n => n
.Path(Infer.Field<OwnerIndex>(ff => ff.mgmt))
.Query(nq => nq
.MultiMatch(m => m
.Fields(f => f
.Field(Infer.Field<OwnerIndex>(ff => ff.owner.market))
.Field(Infer.Field<OwnerIndex>(ff => ff.owner.name))
)
.Operator(Operator.Or)
.Query(searchQuery)
.Fuzziness(Fuzziness.Auto)
)
&& +q.Bool(bq => bq.Filter(ownerFilters))
))
)
).From((skip - 1) * limit)
.Size(limit)
);
return searchResponse.Documents;
}
calling the SearchPropertiesAsync method returns this error messages (truncated for brevity)
....
"index": "owner",
"caused_by": {
"type": "illegal_state_exception",
"reason": "[nested] failed to find nested object under path [property]"
}
....
"index": "property",
"caused_by": {
"type": "illegal_state_exception",
"reason": "[nested] failed to find nested object under path [owner]"
}
Notice that it looks like its trying to perform a nested search of owner. on property index and a nested search of property. on owner index which doesnt exist.
I feel like this should be a very trivial problem but I have been using ElasticSearch for only 4days now and still very new into it.
Is there something I am doing wrongly or is there something I am missing. Have searched the whole internet to even arrive at the solution I have at the moment.
Note that when you executed the nested query only one index at a time, the code works fine but trying to execute on multiple Indices is where my problem lies. Any help will be highly appreciated.
I am using ElasticSearch 7.3.2 and Nest Client 7.3.0.
I don't mind downgrading to a lower version that works.
Apparently, according to the docs
ignore_unmapped
(Optional, boolean) Indicates whether to ignore an unmapped path and not return any documents instead of an error. Defaults to false.
If false, Elasticsearch returns an error if the path is an unmapped field.
You can use this parameter to query multiple indices that may not contain the field path.
So chaining .IgnoreUnmapped(true) on the query body for each of the nested query solved the problem.
Just in case someone else encounter same problem

Can I change this Query into a method?

I am using Entity Framework and have to query a "Project" property a lot, which has a lot of properties which have to be included in the query, making the query bloated:
var project = await this.ProjectRepository.GetRaw().Where(x => x.ID == id).Include(x => x.Leader).Include(x => x.Users).Include(x => x.Firm).FirstOrDefaultAsync();
Please not that GetRaw() returns a IQueryable
Is there a way to construct some helper method where the "include" portion is added, while I dynamically pass in the rest of the query?
something like that:
public async Task<List<Project>> GetProjects(query)
{
return this.ProjectRepository.GetRaw().Include(x => x.Leader).Include(x => x.Users).Include(x => x.Firm) + query;
}
usage something like that:
public ProjectController
{
public void Test()
{
var result = GetProjects.Where(x => x.ID == 0).FirstOrDefaultAsync();
}
}
I think it should look more like this:
public ObjectQuery<Project> GetProjects()
{
return this.ProjectRepository.GetRaw()
.Include(x => x.Leader)
.Include(x => x.Users)
.Include(x => x.Firm);
}
Usage would look like this then:
var result = GetProjects().Where(x => x.ID == 0).FirstOrDefaultAsync();
You can create the expression as parameter and return it as IQueryable<T> so it can be continued by another query.
public IQueryable<Project> GetProjects(Expression<Func<Project, bool> query)
{
return this.ProjectRepository.GetRaw()
.Include(x => x.Leader)
.Include(x => x.Users)
.Include(x => x.Firm)
.Where(query);
}
Usage.
var project = await GetProjects(x => x.ID == 0).FirstOrDefaultAsync();

Telerik MVC - Numeric textbox in grid editor

In my editor for insert telerik show only common textbox (not integer,decimal etc).
this is my view:
#(Html.Telerik().Grid <GamePlayer2>(Model).Name("za11").DataKeys(keys => keys.Add(c => c.Id))
.ToolBar(commands => commands.Insert()
.ButtonType(GridButtonType.ImageAndText))
.DataBinding(dataBinding => dataBinding.Server()
.Insert("PlayerAdd", "Player", new { })
.Update("PlayerUpdate", "Player", new { })
.Delete("PlayerDelete", "Player", new { })
)
.Columns(columns =>
{
columns.Bound(b => b.Name);
columns.Bound(b => b.Price);
columns.Command(commands =>
{
commands.Edit();
commands.Delete();
}).Width(200);
})
.Editable(editing => editing.Mode(GridEditMode.PopUp)))
model:
public class GamePlayer2
{
public int Id { get; set; }
public string Name { get; set; }
[DataType(DataType.Currency)]
public decimal Price{ get; set; }
}
Scripts are registered on _Layout. Where is a problem? Why currency(integer,date etc) textbox won't display for fields?
Make sure you have added the EditorTemplates under the Shared/EditorTemplates folder. Basically they are automatically added if you have used the wizard to create your project. If not create a demo project and copy-paste them from there.

Automapper Question

I have four entities Organization and Address ..for the sake of brevity I have omitted all but the properties in question in my entities and viewmodels
public class Organization : VersionedAuditableEntityWithTypedId<Guid>, IAggregateRoot
{
//Organization related properties are in here
/// <summary>
/// Organization Address
/// </summary>
[NotNull]
public virtual Address Address { get; set; }
}
public class Address : AuditableEntityWithTypedId<Guid>
{
// Address related properties are in here
/// <summary>
/// Region/State
/// </summary>
[NotNull]
public virtual Region Region { get; set; }
}
public class Country : EntityWithTypedId<Guid>
{
/// <summary>
/// Name of country
/// </summary>
[DomainSignature, Length(Max = 50), NotNullNotEmpty, SelectListItemDisplay, SelectListItemValue]
public virtual string Name { get; set; }
}
public class Region : EntityWithTypedId<Guid>
{
/// <summary>
/// Name of region
/// </summary>
[Length(Max = 50), NotNullNotEmpty, SelectListItemDisplay,SelectListItemValue]
public virtual string Name { get; set; }
/// <summary>
/// Country
/// </summary>
[NotNull]
[SuppressMessage("Microsoft.Naming", "CA1721", Justification = "Common naming for type descriptions")]
public virtual Country Country { get; set; }
}
I have two view models
EditOrganizationViewModel
EditAddressViewModel
Within my EditOrganizationViewModel I have the following property
public class EditOrganizationViewModel
{
/// <summary>
/// Organization address
/// </summary>
public EditAddressViewModel Address { get; set; }
}
My EditAddressViewModel has a list of countries
public class EditAddressViewModel
{
/// <summary>
/// List of countries
/// </summary>
public IEnumerable<SelectListItem> Countries { get; set; }
}
My countries property does not map to anything in my address entity but is needed to display a list of countries and it popluated in my service layer.
I perform mapping once in my application as so...
Once for the EditAddressViewModel...
Mapper.CreateMap<Address, EditAddressViewModel>()
.ForMember(dest => dest.LabelForAddress1, opt => opt.Ignore())
.ForMember(dest => dest.LabelForAddress2, opt => opt.Ignore())
.ForMember(dest => dest.LabelForAddress3, opt => opt.Ignore())
.ForMember(dest => dest.LabelForAddressType, opt => opt.Ignore())
.ForMember(dest => dest.LabelForCity, opt => opt.Ignore())
.ForMember(dest => dest.LabelForCountry, opt => opt.Ignore())
.ForMember(dest => dest.LabelForPostcode, opt => opt.Ignore())
.ForMember(dest => dest.LabelForRegion, opt => opt.Ignore())
.ForMember(dest => dest.Types, opt => opt.Ignore())
.ForMember(dest => dest.Countries, opt => opt.Ignore())
.ForMember(dest => dest.Regions, opt => opt.Ignore())
.ForMember(dest => dest.Country, opt => opt.MapFrom(src => src.Region.Country.Id))
.ForMember(dest => dest.Region, opt => opt.MapFrom(src => src.Region.Id))
.ForMember(dest => dest.Type, opt => opt.MapFrom(src => src.Type.Id))
Mapper.CreateMap<EditAddressViewModel, Address>();
and once for the EditOrganizationViewModel..
Mapper.CreateMap<Organization, EditOrganizationViewModel>()
.ForMember(dest => dest.LabelForOrganizationName, opt => opt.Ignore())
.ForMember(dest => dest.LabelForTeamSport, opt => opt.Ignore())
.AfterMap(
(organization, viewModel) =>
MapListItemToViewModel(organization.Phones, viewModel.Phone,
p => p.Type.Name == Resources.BusinessPhone))
.AfterMap(
(organization, viewModel) =>
MapListItemToViewModel(organization.Phones, viewModel.Fax,
p => p.Type.Name == Resources.FaxPhone))
.AfterMap(
(organization, viewModel) =>
MapListItemToViewModel(organization.EmailAddresses, viewModel.EmailAddress,
p => p.Type.Name == Resources.PrimaryEmailAddress));
I build my view model as below
public EditOrganizationViewModel CreateOrganizationFormViewModel()
{
// Create the organization view model
var viewModel = new EditOrganizationViewModel();
// Load the organization and bind to view model
var organization = _organizationRepository.FindAll().FirstOrDefault();
if (organization == null) organization = new Organization();
// Business Address
viewModel.Address =
AddressManagementService.Instance.CreateEditAddressViewModel(Resources.BusinessAddress);
**At this point my countries list is populated**
// Map the organization to the view model
Mapper.Map(organization, viewModel);
**My countries property is now null after Mapper.Map is called**
}
The problem arises when I call Mapper.Map(organization, viewModel). The countries list is now null. I am not sure how setup my mapping configurations to preserve my list of coutries. I have tried using a valueresolver but it only gets called if the address of the organization is not null. I would hate to have to really flatten my organizationviewmodel and include the address information. Any help would be greatly appreciated.
if the or
either of 2 things should work...
for that field in the mapping, instead of Ignore() try UseDestinationValue()
populate the country list after the map is executed (reverse the order of the last 2 lines of your function)

Server-side Validation using MVC 3

I'm building an ASP.NET MVC3 app. I have 2 views;
List item contains a grid
details view consists of a drop down list (combobox)
I have a requirement to alert the user at the details view when they try to select item was previously selected in the list view. In other words, the grid should contain unique items
What is the best way to implement a server-side business logic validation?
Model:
public class AllocatedResource
{
public virtual Project Project { get; set; }
public virtual DateTime StartDate { get; set; }
public virtual DateTime EndDate { get; set; }
}
List View:
#(Html.Telerik().Grid(Model.AllocatedResources)
.Name("gridAllocatedProject")
.DataKeys(keys =>{keys.Add(p => p.Id);})
.Columns(columns =>
{
columns.Bound(p => p.Id).Visible(false);
columns.Bound(p => p.Project.Name);
columns.Bound(p => p.Project.ProjectManager).Title("Project Manager");
columns.Bound(p => p.StartDate).Width(80).Format("{0:d}");
columns.Bound(p => p.EndDate).Width(80).Format("{0:d}");
})
Details View:
#Html.Label("Project: ")
#(Html.Telerik().ComboBox().Name("Project")
.BindTo(new SelectList(Model.AllProjects, "Id", "Name"))
.Value(Model.AllocatedResource.Project.Id.ToString()))
#Html.Label("Start Date: ")
#(Html.Telerik().DatePicker().Name("StartDate")
.Value(Model.AllocatedResource.StartDate))
#Html.Label("End Date: ")
#(Html.Telerik().DatePicker().Name("EndDate")
.Value(Model.AllocatedResource.EndDate))

Resources