Entity Framework, Code First and Full Text Search - linq

I realize that a lot of questions have been asked relating to full text search and Entity Framework, but I hope this question is a bit different.
I am using Entity Framework, Code First and need to do a full text search. When I need to perform the full text search, I will typically have other criteria/restrictions as well - like skip the first 500 rows, or filter on another column, etc.
I see that this has been handled using table valued functions - see http://sqlblogcasts.com/blogs/simons/archive/2008/12/18/LINQ-to-SQL---Enabling-Fulltext-searching.aspx. And this seems like the right idea.
Unfortunately, table valued functions are not supported until Entity Framework 5.0 (and even then, I believe, they are not supported for Code First).
My real question is what are the suggestions for the best way to handle this, both for Entity Framework 4.3 and Entity Framework 5.0. But to be specific:
Other than dynamic SQL (via System.Data.Entity.DbSet.SqlQuery, for example), are there any options available for Entity Framework 4.3?
If I upgrade to Entity Framework 5.0, is there a way I can use table valued functions with code first?
Thanks,
Eric

Using interceptors introduced in EF6, you could mark the full text search in linq and then replace it in dbcommand as described in http://www.entityframework.info/Home/FullTextSearch:
public class FtsInterceptor : IDbCommandInterceptor
{
private const string FullTextPrefix = "-FTSPREFIX-";
public static string Fts(string search)
{
return string.Format("({0}{1})", FullTextPrefix, search);
}
public void NonQueryExecuting(DbCommand command, DbCommandInterceptionContext<int> interceptionContext)
{
}
public void NonQueryExecuted(DbCommand command, DbCommandInterceptionContext<int> interceptionContext)
{
}
public void ReaderExecuting(DbCommand command, DbCommandInterceptionContext<DbDataReader> interceptionContext)
{
RewriteFullTextQuery(command);
}
public void ReaderExecuted(DbCommand command, DbCommandInterceptionContext<DbDataReader> interceptionContext)
{
}
public void ScalarExecuting(DbCommand command, DbCommandInterceptionContext<object> interceptionContext)
{
RewriteFullTextQuery(command);
}
public void ScalarExecuted(DbCommand command, DbCommandInterceptionContext<object> interceptionContext)
{
}
public static void RewriteFullTextQuery(DbCommand cmd)
{
string text = cmd.CommandText;
for (int i = 0; i < cmd.Parameters.Count; i++)
{
DbParameter parameter = cmd.Parameters[i];
if (parameter.DbType.In(DbType.String, DbType.AnsiString, DbType.StringFixedLength, DbType.AnsiStringFixedLength))
{
if (parameter.Value == DBNull.Value)
continue;
var value = (string)parameter.Value;
if (value.IndexOf(FullTextPrefix) >= 0)
{
parameter.Size = 4096;
parameter.DbType = DbType.AnsiStringFixedLength;
value = value.Replace(FullTextPrefix, ""); // remove prefix we added n linq query
value = value.Substring(1, value.Length - 2);
// remove %% escaping by linq translator from string.Contains to sql LIKE
parameter.Value = value;
cmd.CommandText = Regex.Replace(text,
string.Format(
#"\[(\w*)\].\[(\w*)\]\s*LIKE\s*#{0}\s?(?:ESCAPE N?'~')",
parameter.ParameterName),
string.Format(#"contains([$1].[$2], #{0})",
parameter.ParameterName));
if (text == cmd.CommandText)
throw new Exception("FTS was not replaced on: " + text);
text = cmd.CommandText;
}
}
}
}
}
static class LanguageExtensions
{
public static bool In<T>(this T source, params T[] list)
{
return (list as IList<T>).Contains(source);
}
}
For example, if you have class Note with FTS-indexed field NoteText:
public class Note
{
public int NoteId { get; set; }
public string NoteText { get; set; }
}
and EF map for it
public class NoteMap : EntityTypeConfiguration<Note>
{
public NoteMap()
{
// Primary Key
HasKey(t => t.NoteId);
}
}
and context for it:
public class MyContext : DbContext
{
static MyContext()
{
DbInterception.Add(new FtsInterceptor());
}
public MyContext(string nameOrConnectionString) : base(nameOrConnectionString)
{
}
public DbSet<Note> Notes { get; set; }
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Configurations.Add(new NoteMap());
}
}
you can have quite simple syntax to FTS query:
class Program
{
static void Main(string[] args)
{
var s = FtsInterceptor.Fts("john");
using (var db = new MyContext("CONNSTRING"))
{
var q = db.Notes.Where(n => n.NoteText.Contains(s));
var result = q.Take(10).ToList();
}
}
}
That will generate SQL like
exec sp_executesql N'SELECT TOP (10)
[Extent1].[NoteId] AS [NoteId],
[Extent1].[NoteText] AS [NoteText]
FROM [NS].[NOTES] AS [Extent1]
WHERE contains([Extent1].[NoteText], #p__linq__0)',N'#p__linq__0 char(4096)',#p__linq__0='(john)
Please notice that you should use local variable and cannot move FTS wrapper inside expression like
var q = db.Notes.Where(n => n.NoteText.Contains(FtsInterceptor.Fts("john")));

I have found that the easiest way to implement this is to setup and configure full-text-search in SQL Server and then use a stored procedure. Pass your arguments to SQL, allow the DB to do its job and return either a complex object or map the results to an entity. You don't necessarily have to have dynamic SQL, but it may be optimal. For example, if you need paging, you could pass in PageNumber and PageSize on every request without the need for dynamic SQL. However, if the number of arguments fluctuates per query, it will be the optimal solution.

As the other guys mentioned, I would say start using Lucene.NET
Lucene has a pretty high learning curve, but I found an wrapper for it called "SimpleLucene", that can be found on CodePlex
Let me quote a couple of codeblocks from the blog to show you how easy it is to use. I've just started to use it, but got the hang of it really fast.
First, get some entities from your repository, or in your case, use Entity Framework
public class Repository
{
public IList<Product> Products {
get {
return new List<Product> {
new Product { Id = 1, Name = "Football" },
new Product { Id = 2, Name = "Coffee Cup"},
new Product { Id = 3, Name = "Nike Trainers"},
new Product { Id = 4, Name = "Apple iPod Nano"},
new Product { Id = 5, Name = "Asus eeePC"},
};
}
}
}
The next thing you want to do is create an index-definition
public class ProductIndexDefinition : IIndexDefinition<Product> {
public Document Convert(Product p) {
var document = new Document();
document.Add(new Field("id", p.Id.ToString(), Field.Store.YES, Field.Index.NOT_ANALYZED));
document.Add(new Field("name", p.Name, Field.Store.YES, Field.Index.ANALYZED));
return document;
}
public Term GetIndex(Product p) {
return new Term("id", p.Id.ToString());
}
}
and create an search index for it.
var writer = new DirectoryIndexWriter(
new DirectoryInfo(#"c:\index"), true);
var service = new IndexService();
service.IndexEntities(writer, Repository().Products, ProductIndexDefinition());
So, you now have an search-able index. The only remaining thing to do is.., searching! You can do pretty amazing things, but it can be as easy as this: (for greater examples see the blog or the documentation on codeplex)
var searcher = new DirectoryIndexSearcher(
new DirectoryInfo(#"c:\index"), true);
var query = new TermQuery(new Term("name", "Football"));
var searchService = new SearchService();
Func<Document, ProductSearchResult> converter = (doc) => {
return new ProductSearchResult {
Id = int.Parse(doc.GetValues("id")[0]),
Name = doc.GetValues("name")[0]
};
};
IList<Product> results = searchService.SearchIndex(searcher, query, converter);

The example here http://www.entityframework.info/Home/FullTextSearch is not complete solution. You will need to look into understand how the full text search works. Imagine you have a search field and the user types 2 words to hit search. The above code will throw an exception. You need to do pre-processing on the search phrase first to pass it to the query by using logical AND or OR.
for example your search phrase is "blah blah2" then you need to convert this into:
var searchTerm = #"\"blah\" AND/OR \"blah2\" ";
Complete solution would be:
value = Regex.Replace(value, #"\s+", " "); //replace multiplespaces
value = Regex.Replace(value, #"[^a-zA-Z0-9 -]", "").Trim();//remove non-alphanumeric characters and trim spaces
if (value.Any(Char.IsWhiteSpace))
{
value = PreProcessSearchKey(value);
}
public static string PreProcessSearchKey(string searchKey)
{
var splitedKeyWords = searchKey.Split(null); //split from whitespaces
// string[] addDoubleQuotes = new string[splitedKeyWords.Length];
for (int j = 0; j < splitedKeyWords.Length; j++)
{
splitedKeyWords[j] = $"\"{splitedKeyWords[j]}\"";
}
return string.Join(" AND ", splitedKeyWords);
}
this methods uses AND logic operator. You might pass that as an argument and use the method for both AND or OR operators.
You must escape none-alphanumeric characters otherwise it would throw exception when a user enters alpha numeric characters and you have no server site model level validation in place.

I recently had a similar requirement and ended up writing an IQueryable extension specifically for Microsoft full text index access, its available here IQueryableFreeTextExtensions

Related

Calling Dynamics Web API with Entity metadata early binding

I would like to consume my organizations dynamics oData endpoint but with early bound classes. However, there are a lot of early bound tools out there and I wanted to know which one provides the best developer experience/least resistance?
For example, there is this one:
https://github.com/daryllabar/DLaB.Xrm.XrmToolBoxTools
https://github.com/yagasoft/DynamicsCrm-CodeGenerator
and so on. Is there a developer preference/method out there?
Early bound classes are for use with the Organization Service which is a SOAP service. The normal way to generate those classes is using CrmSvcUtil.
OData can be used in Organization Data Service or Web API, but those don't have Early Bound classes.
Further reading: Introducing the Microsoft Dynamics 365 web services
It's not impossible to use with standard SOAP Early bound class. We just have to be creative. If we work just with basic attributes (fields, not relationships, ecc) it seems possible. For example. for create and update, OData will not accept the entire early bounded class, just pass the attibutes:
class Program
{
static void Main(string[] args)
{
string token = System.Threading.Tasks.Task.Run(() => GetToken()).Result;
CRMWebAPI dynamicsWebAPI = new CRMWebAPI("https:/ORG.api.crm4.dynamics.com/api/data/v9.1/",
token);
CRMGetListOptions listOptions = new CRMGetListOptions
{
Select = new string[] { "EntitySetName" },
Filter = "LogicalName eq 'contact'"
};
dynamic entityDefinitions = dynamicsWebAPI.GetList<ExpandoObject>("EntityDefinitions", listOptions).Result;
Contact contact = new Contact
{
FirstName = "Felipe",
LastName = "Test",
MobilePhone = "38421254"
};
dynamic ret = System.Threading.Tasks.Task.Run(async () => await dynamicsWebAPI.Create(entityDefinitions.List[0].EntitySetName, KeyPairValueToObject(contact.Attributes))).Result;
}
public static async Task<string> GetToken()
{
string api = "https://ORG.api.crm4.dynamics.com/";
ClientCredential credential = new ClientCredential("CLIENT_ID", "CLIENT_SECRET");
AuthenticationContext authenticationContext = new AuthenticationContext("https://login.microsoftonline.com/commom/oauth2/authorize");
return authenticationContext.AcquireTokenAsync(api, credential).Result.AccessToken;
}
public static object KeyPairValueToObject(AttributeCollection keyValuePairs)
{
dynamic expando = new ExpandoObject();
var obj = expando as IDictionary<string, object>;
foreach (var keyValuePair in keyValuePairs)
obj.Add(keyValuePair.Key, keyValuePair.Value);
return obj;
}
}
It's a simple approach and I didn't went further.
Maybe we have to serealize other objects as OptionSets, DateTime (pass just the string) and EntityReferences but this simple test worked fine to me. I'm using Xrm.Tools.WebAPI and Microsoft.IdentityModel.Clients.ActiveDirectory. Maybe it's a way.
[Edit]
And so I decided to go and created a not well tested method to cast the attributes. Problems: We have to follow OData statments to use the API. To update/create an entity reference we can use this to reference https://www.inogic.com/blog/2016/02/set-values-of-all-data-types-using-web-api-in-dynamics-crm/
So
//To EntityReference
entityToUpdateOrCreate["FIELD_SCHEMA_NAME#odata.bind"] = "/ENTITY_SET_NAME(GUID)";
So, it's the Schema name, not field name. If you use CamelCase when set you fields name you'll have a problem where. We can resolve that with a (to that cute) code
public static object EntityToObject<T>(T entity) where T : Entity
{
dynamic expando = new ExpandoObject();
var obj = expando as IDictionary<string, object>;
foreach (var keyValuePair in entity.Attributes)
{
obj.Add(GetFieldName(entity, keyValuePair), CastEntityAttibutesValueOnDynamicObject(keyValuePair.Value));
}
return obj;
}
public static object CastEntityAttibutesValueOnDynamicObject(object attributeValue)
{
if (attributeValue.GetType().Name == "EntityReference")
{
CRMGetListOptions listOptions = new CRMGetListOptions
{
Select = new string[] { "EntitySetName" },
Filter = $"LogicalName eq '{((EntityReference)attributeValue).LogicalName}'"
};
dynamic entitySetName = dynamicsWebAPI.GetList<ExpandoObject>("EntityDefinitions", listOptions).Result.List[0];
return $"/{entitySetName.EntitySetName}({((EntityReference)attributeValue).Id})";
}
else if (attributeValue.GetType().Name == "OptionSetValue")
{
return ((OptionSetValue)attributeValue).Value;
}
else if (attributeValue.GetType().Name == "DateTime")
{
return ((DateTime)attributeValue).ToString("yyyy-MM-dd");
}
else if (attributeValue.GetType().Name == "Money")
{
return ((Money)attributeValue).Value;
}
else if (attributeValue.GetType().Name == "AliasedValue")
{
return CastEntityAttibutesValueOnDynamicObject(((AliasedValue)attributeValue).Value);
}
else
{
return attributeValue;
}
}
public static string GetFieldName<T>(T entity, KeyValuePair<string, object> keyValuePair) where T : Entity
{
switch (keyValuePair.Value.GetType().Name)
{
case "EntityReference":
var entityNameList = System.Threading.Tasks.Task.Run(async () => await dynamicsWebAPI.GetEntityDisplayNameList()).Result;
var firstEntity = entityNameList.Where(x => x.LogicalName == entity.LogicalName).FirstOrDefault();
var attrNameList = System.Threading.Tasks.Task.Run(async () => await dynamicsWebAPI.GetAttributeDisplayNameList(firstEntity.MetadataId)).Result;
return attrNameList.Where(x => x.LogicalName == keyValuePair.Key).Single().SchemaName + "#odata.bind";
case "ActivityParty":
throw new NotImplementedException(); //TODO
default:
return keyValuePair.Key;
}
}
Please, note that this approach do not seems fast or good in anyway. It's better if you have all this values as static so we can save some fetches
[Edit 2]
I just found on XRMToolBox a plugin called "Early bound generator for Web API" and it seems to be the best option. Maybe you should give it a try if you're still curious about that. I guess its the best approach.
The final code is this:
static void Main(string[] args)
{
string token = Task.Run(() => GetToken()).Result;
dynamicsWebAPI = new CRMWebAPI("https://ORG.api.crm4.dynamics.com/api/data/v9.1/",
token);
Contact contact = new Contact
{
FirstName = "Felipe",
LastName = "Test",
MobilePhone = "38421254",
new_Salutation = new EntityReference(new_salutation.EntitySetName, new Guid("{BFA27540-7BB9-E611-80EE-FC15B4281C8C}")),
BirthDate = new DateTime(1993, 04, 14),
};
dynamic ret = Task.Run(async () => await dynamicsWebAPI.Create(Contact.EntitySetName, contact.ToExpandoObject())).Result;
Contact createdContact = dynamicsWebAPI.Get<Contact>(Contact.EntitySetName, ret, new CRMGetListOptions
{
Select = new string[] { "*" }
}).Result;
}
and you have to change the ToExpandoObject on Entity.cs class (generated by the plugin)
public ExpandoObject ToExpandoObject()
{
dynamic expando = new ExpandoObject();
var expandoObject = expando as IDictionary<string, object>;
foreach (var attributes in Attributes)
{
if (attributes.Key == GetIdAttribute())
{
continue;
}
var value = attributes.Value;
var key = attributes.Key;
if (value is EntityReference entityReference)
{
value = $"/{entityReference.EntitySetName}({entityReference.EntityId})";
}
else
{
key = key.ToLower();
if (value is DateTime dateTimeValue)
{
var propertyForAttribute = GetPublicInstanceProperties().FirstOrDefault(x =>
x.Name.Equals(key, StringComparison.InvariantCultureIgnoreCase));
if (propertyForAttribute != null)
{
var onlyDateAttr = propertyForAttribute.GetCustomAttribute<OnlyDateAttribute>();
if (onlyDateAttr != null)
{
value = dateTimeValue.ToString(OnlyDateAttribute.Format);
}
}
}
}
expandoObject.Add(key, value);
}
return (ExpandoObject)expandoObject;
}
Links:
https://github.com/davidyack/Xrm.Tools.CRMWebAPI
https://www.xrmtoolbox.com/plugins/crm.webApi.earlyBoundGenerator/
We currently use XrmToolkit which has it's own version of early binding called ProxyClasses but will allow you to generate early binding using the CRM Service Utility (CrmSvcUtil). It does a lot more than just early binding which is why we use it on all of our projects but the early binding features alone would have me sold on it. in order to regenerate an entity definition all you do is right click the cs file in visual studio and select regenerate and it is done in a few seconds.
For my first 3 years of CRM development I used the XrmToolbox "Early Bound Generator" plugin which is really helpful as well.

How to select multiple class properties in LINQ Expression?

If I have a class like this
`
class Person
{
public string First;
public string Last;
public bool IsMarried;
public int Age;
}`
Then how can I write a LINQ Expression where I could select properties of a Person. I want to do something like this (user can enter 1..n properties)
SelectData<Person>(x=>x.First, x.Last,x.Age);
What would be the input expression of my SelectData function ?
SelectData(Expression<Func<TEntity, List<string>>> selector); ?
EDIT
In my SelectData function I want to extract property names and then generate SELECT clause of my SQL Query dynamically.
SOLUTION
Ok, so what I have done is to have my SelectData as
public IEnumerable<TEntity> SelectData(Expression<Func<TEntity, object>> expression)
{
NewExpression body = (NewExpression)expression.Body;
List<string> columns = new List<string>();
foreach(var arg in body.Arguments)
{
var exp = (MemberExpression)arg;
columns.Add(exp.Member.Name);
}
//build query
And to use it I call it like this
ccc<Person>().SelectData(x => new { x.First, x.Last, x.Age });
Hopefully it would help someone who is looking :)
Thanks,
IY
I think it would be better to use delegates instead of Reflection. Apart from the fact that delegates will be faster, the compiler will complain if you try to fetch property values that do not exist. With reflection you won't find errors until run time.
Luckily there is already something like that. it is implemented as an extension function of IEnumerable, and it is called Select (irony intended)
I think you want something like this:
I have a sequence of Persons, and I want you to create a Linq
statement that returns per Person a new object that contains the
properties First and Last.
Or:
I have a sequence of Persns and I want you to create a Linq statement
that returns per Person a new object that contains Age, IsMarried,
whether it is an adult and to make it difficult: one Property called
Name which is a combination of First and Last
The function SelectData would be something like this:
IEnumerable<TResult> SelectData<TSource, TResult>(this IEnumerable<TSource> source,
Func<TSource, TResult> selector)
{
return source.Select(selector);
}
Usage:
problem 1: return per Person a new object that contains the
properties First and Last.
var result = Persons.SelectData(person => new
{
First = person.First,
Last = person.Last,
});
problem 2: return per Person a new object that contains Age, IsMarried, whether he is an adult and one Property called Name which is a combination
of First and Last
var result = Persons.SelectData(person => new
{
Age = person.Name,
IsMarried = person.IsMarried,
IsAdult = person.Age > 21,
Name = new
{
First = person.First,
Last = person.Last,
},
});
Well let's face it, your SelectData is nothing more than Enumerable.Select
You could of course create a function where you'd let the caller provide a list of properties he wants, but (1) that would limit his possibilities to design the end result and (2) it would be way more typing for him to call the function.
Instead of:
.Select(p => new
{
P1 = p.Property1,
P2 = p.Property2,
}
he would have to type something like
.SelectData(new List<Func<TSource, TResult>()
{
p => p.Property1, // first element of the property list
p -> p.Property2, // second element of the property list
}
You won't be able to name the returned properties, you won't be able to combine several properties into one:
.Select(p => p.First + p.Last)
And what would you gain by it?
Highly discouraged requirement!
You could achive similar result using Reflection and Extension Method
Model:
namespace ConsoleApplication2
{
class Person
{
public string First { get; set; }
public string Last { get; set; }
public bool IsMarried { get; set; }
public int Age { get; set; }
}
}
Service:
using System.Collections.Generic;
using System.Linq;
namespace Test
{
public static class Service
{
public static IQueryable<IQueryable<KeyValuePair<string, object>>> SelectData<T>(this IQueryable<T> queryable, string[] properties)
{
var queryResult = new List<IQueryable<KeyValuePair<string, object>>>();
foreach (T entity in queryable)
{
var entityProperties = new List<KeyValuePair<string, object>>();
foreach (string property in properties)
{
var value = typeof(T).GetProperty(property).GetValue(entity);
var entityProperty = new KeyValuePair<string, object>(property, value);
entityProperties.Add(entityProperty);
}
queryResult.Add(entityProperties.AsQueryable());
}
return queryResult.AsQueryable();
}
}
}
Usage:
using System;
using System.Collections.Generic;
using System.Linq;
namespace Test
{
class Program
{
static void Main(string[] args)
{
var list = new List<Person>()
{
new Person()
{
Age = 18,
First = "test1",
IsMarried = false,
Last = "test2"
},
new Person()
{
Age = 40,
First = "test3",
IsMarried = true,
Last = "test4"
}
};
var queryableList = list.AsQueryable();
string[] properties = { "Age", "Last" };
var result = queryableList.SelectData(properties);
foreach (var element in result)
{
foreach (var property in element)
{
Console.WriteLine($"{property.Key}: {property.Value}");
}
}
Console.ReadKey();
}
}
}
Result:
Age: 18
Last: test2
Age: 40
Last: test4

Entity framework 6 code first Many to many insert slow

I am trying to find a way to improve insert performances with the following code (please, read my questions after the code block):
//Domain classes
[Table("Products")]
public class Product
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int Id { get; set; }
public string Sku { get; set; }
[ForeignKey("Orders")]
public virtual ICollection<Order> Orders { get; set; }
public Product()
{
Orders = new List<Order>();
}
}
[Table("Orders")]
public class Order
{
[Key]
[DatabaseGenerated(DatabaseGeneratedOption.Identity)]
public int Id { get; set; }
public string Title { get; set; }
public decimal Total { get; set; }
[ForeignKey("Products")]
public virtual ICollection<Product> Products { get; set; }
public Order()
{
Products = new List<Product>();
}
}
//Data access
public class MyDataContext : DbContext
{
public MyDataContext()
: base("MyDataContext")
{
Configuration.LazyLoadingEnabled = true;
Configuration.ProxyCreationEnabled = true;
Database.SetInitializer(new CreateDatabaseIfNotExists<MyDataContext>());
}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.Entity<Product>().ToTable("Products");
modelBuilder.Entity<Order>().ToTable("Orders");
}
}
//Service layer
public interface IServices<T, K>
{
T Create(T item);
T Read(K key);
IEnumerable<T> ReadAll(Expression<Func<IEnumerable<T>, IEnumerable<T>>> pre);
T Update(T item);
void Delete(K key);
void Save();
void Dispose();
void BatchSave(IEnumerable<T> list);
void BatchUpdate(IEnumerable<T> list, Action<UpdateSpecification<T>> spec);
}
public class BaseServices<T, K> : IDisposable, IServices<T, K> where T : class
{
protected MyDataContext Context;
public BaseServices()
{
Context = new MyDataContext();
}
public T Create(T item)
{
T created;
created = Context.Set<T>().Add(item);
return created;
}
public void Delete(K key)
{
var item = Read(key);
if (item == null)
return;
Context.Set<T>().Attach(item);
Context.Set<T>().Remove(item);
}
public T Read(K key)
{
T read;
read = Context.Set<T>().Find(key);
return read;
}
public IEnumerable<T> ReadAll(Expression<Func<IEnumerable<T>, IEnumerable<T>>> pre)
{
IEnumerable<T> read;
read = Context.Set<T>().ToList();
read = pre.Compile().Invoke(read);
return read;
}
public T Update(T item)
{
Context.Set<T>().Attach(item);
Context.Entry<T>(item).CurrentValues.SetValues(item);
Context.Entry<T>(item).State = System.Data.Entity.EntityState.Modified;
return item;
}
public void Save()
{
Context.SaveChanges();
}
}
public interface IOrderServices : IServices<Order, int>
{
//custom logic goes here
}
public interface IProductServices : IServices<Product, int>
{
//custom logic goes here
}
//Web project's controller
public ActionResult TestCreateProducts()
{
//Create 100 new rest products
for (int i = 0; i < 100; i++)
{
_productServices.Create(new Product
{
Sku = i.ToString()
});
}
_productServices.Save();
var products = _productServices.ReadAll(r => r); //get a list of saved products to add them to orders
var random = new Random();
var orders = new List<Order>();
var count = 0;
//Create 3000 orders
for (int i = 1; i <= 3000; i++)
{
//Generate a random list of products to attach to the current order
var productIds = new List<int>();
var x = random.Next(1, products.Count() - 1);
for (int j = 0; j < x; j++)
{
productIds.Add(random.Next(products.Min(r => r.Id), products.Max(r => r.Id)));
}
//Create the order
var order = new Order
{
Title = "Order" + i,
Total = i,
Products = products.Where(p => productIds.Contains(p.Id))
};
orders.Add(order);
}
_orderServices.CreateRange(orders);
_orderServices.Save();
return RedirectToAction("Index");
}
This code works fine but is very VERY slow when the SaveChanges is executed.
Behind the scene, the annotations on the domain objects creates all the relationships needed: a OrderProducts table with the proper foreign keys are automatically created and the inserts are being done by EF properly.
I've tried many things with bulk inserts using EntityFramework.Utilities, SqlBulkCopy, etc... but none worked.
Is there a way to achieve this?
Understand this is only for testing purposes and my goal is to optimize the best I can any operations in our softwares using EF.
Thanks!
Just before you do your inserts disable your context's AutoDetectChangesEnabled (by setting it to false). Do your inserts and then set the AutoDetectChangesEnabled back to true e.g.;
try
{
MyContext.Configuration.AutoDetectChangesEnabled = false;
// do your inserts updates etc..
}
finally
{
MyContext.Configuration.AutoDetectChangesEnabled = true;
}
You can find more information on what this is doing here
I see two reasons why your code is slow.
Add vs. AddRange
You add entity one by one using the Create method.
You should always use AddRange over Add. The Add method will try to DetectChanges every time the add method is invoked while AddRange only once.
You should add a "CreateRange" method in your code.
public IEnumerable<T> CreateRange(IEnumerable<T> list)
{
return Context.Set<T>().AddRange(list);
}
var products = new List<Product>();
//Create 100 new rest products
for (int i = 0; i < 100; i++)
{
products.Add(new Product { Sku = i.ToString() });
}
_productServices.CreateRange(list);
_productServices.Save();
Disabling / Enabling the property AutoDetectChanges also work as #mark_h proposed, however personally I don't like this kind of solution.
Database Round Trip
A database round trip is required for every record to add, modify or delete. So if you insert 3,000 records, then 3,000 database round trip will be required which is VERY slow.
You already tried EntityFramework.BulkInsert or SqlBulkCopy, which is great. I recommend you first to try them again using the "AddRange" fix to see the newly performance.
Here is a biased comparison of library supporting BulkInsert for EF:
Entity Framework - Bulk Insert Library Reviews & Comparisons
Disclaimer: I'm the owner of the project Entity Framework Extensions
This library allows you to BulkSaveChanges, BulkInsert, BulkUpdate, BulkDelete and BulkMerge within your Database.
It supports all inheritances and associations.
// Easy to use
public void Save()
{
// Context.SaveChanges();
Context.BulkSaveChanges();
}
// Easy to customize
public void Save()
{
// Context.SaveChanges();
Context.BulkSaveChanges(bulk => bulk.BatchSize = 100);
}
EDIT: Added answer to sub question
An entity object cannot be referenced by multiple instances of
IEntityChangeTracker
The issue happens because you use two different DbContext. One for the product and one for order.
You may find a better answer than mine in a different thread like this answer.
The Add method successfully attach the product, subsequent call of the same product doesn't throw an error because it's the same product.
The AddRange method, however, attach the product multiple time since it's not come from the same context, so when Detect Changes is called, he doesn't know how to handle it.
One way to fix it is by re-using the same context
var _productServices = new BaseServices<Product, int>();
var _orderServices = new BaseServices<Order, int>(_productServices.Context);
While it may not be elegant, the performance will be improved.

Attempting to perform complicated anonymous methods is failing

Update (Stupidity Fail)
So then, in all of my convoluted formula code, I neglected the fundamental principles of C#.
Methods may return a value.
static dynamic Construct<T>(T expression){
return expression;
}
Then just use that, instead of a variable ...
Method = Construct<Action<Context, string, int>>(
(context, key, change) =>
{
context.Saved[key] += change;
Console.WriteLine("{0}'s saved value of {1} was changed by {2}, resulting in {3}",
context.Name, key, change, context.Saved[key]);
}
)
I have a situation where I need to call upon methods that don't exist as compiled methods, but rather need to be able to accept an array of parameters and execute as an anonymous function. I thought I had it worked out, but I am running into an issue with the following..
public static IDictionary<string, Function> Expressions =
new Dictionary<string, Function> {
{
"Increase [X] by value of [Y]",
new Function {
Name = "Increase [X] by [Y]",
Parameters = 2,
Types = new List<Type>{
typeof(Param),
typeof(Param)
},
Method = (Expression<Func<Context, Param, Param, bool>>)
((context, x, y) => {
Console.WriteLine("test"); // this is where I need to do stuff...
})
}
}
};
I am being told that a Method name is expected on this. The problem is that Context will be passed in by the object that takes the function and runs its method, because the Context object cannot be pre-bound (it has to be late bound). So basically I package up the trailing 2 parameters (Param) and (Param) in this case and create a function to execute against them.
The database stores those parameters, and then invokes the method passing in the appropriate Context as the first parameter by using Compile().DynamicInvoke(object[] params).
Can anyone give me a hand here as to why I cannot put any kind of logic in between my { }?
UPDATE
Okay, since I've been told this example is unclear, here is an entire program running start to finish that illustrates what I am trying to accomplish.
public class Program {
static void Main(string[] args) {
// simple object stored in database.
var ctx = new Context {
Name = "Ciel",
Saved = new Dictionary<string, int> {
{ "First", 10 },
{ "Second", 20 }
}
};
// simple object stored in database.
var rule = new Rule {
Equations = new List<Equation> {
new Equation {
Parameters = new List<object>{
"First",
5
},
Name = "Increase [X] by value of [Y]"
}
}
};
// =======================================
// runtime environment!!!
// =======================================
var method = Evaluations.Expressions[rule.Equations[0].Name].Method;
var parameters = rule.Equations[0].Parameters;
// insert the specific context as the first parameter.
parameters.Insert(0, ctx);
method.DynamicInvoke(parameters.ToArray());
Console.ReadLine();
}
}
public class Function {
public string Name { get; set; }
public dynamic Method { get; set; }
}
public class Equation {
public string Name { get; set; }
// these objects will be simple enough to serialize.
public IList<object> Parameters { get; set; }
public Function Function { get; set; }
}
public class Context {
public string Name { get; set; }
// this is a crude example, but it serves the demonstration purposes.
public IDictionary<string, int> Saved { get; set; }
}
public class Rule {
// again, a crude example.
public IList<Equation> Equations { get; set; }
}
public static class Evaluations {
static Action<Context, string, int> expr = (context, key, change) =>
{
context.Saved[key] += change;
Console.WriteLine("{0}'s saved value of {1} was changed by {2}, resulting in {3}",
context.Name, key, change, context.Saved[key]);
};
public static IDictionary<string, Function> Expressions =
new Dictionary<string, Function> {
{
"Increase [X] by value of [Y]",
new Function {
Name = "Increase [X] by [Y]",
Method = expr
}
}
};
}
Four problems:
You're trying to create an expression tree from a lambda expression with a statement body (i.e. braces). C# doesn't allow this - you can only convert a statement lambda into a delegate, not an expression tree
Your lambda body doesn't return a Boolean value
You're trying to call an Expression<Func<Context, Param, Param, bool>> as if it were a method with a bool parameter. It's not at all clear what you're trying to do there.
Even if the third point were valid, I suspect you'd need more brackets.
If you refactor your code to make it a little more readable and manageable, you'll probably be well on your way to solving your problem. Rather than having one mammoth C# statement with a single semicolon, split it up into several lines. Something like this:
public static Dictionary<string, Function> Expressions = getExpressions();
private static Dictionary<string, Function> getExpressions()
{
var method = (Expression<Func<Context, Param, Param, bool>>)
((context, x, y) => {
Console.WriteLine("test"); // this is where I need to do stuff...
})(true);
var func = new Function()
{
Name = "Increase [X] by [Y]",
Parameters = 2,
Types = new List<Type>
{
typeof(Param),
typeof(Param)
},
Method = method
};
var dict = new Dictionary<string, Function>();
dict["Increase [X] by value of [Y]"] = func;
return dict;
}
Note: my syntax could be incorrect, but you get the general idea.

Why predicate isn't filtering when building it via reflection

I'm building a rather large filter based on an SearchObject that has 50+ fields that can be searched.
Rather than building my where clause for each one of these individually I thought I'd use some slight of hand and try building custom attribute suppling the necessary information and then using reflection to build out each of my predicate statements (Using LinqKit btw). Trouble is, that the code finds the appropriate values in the reflection code and successfully builds a predicate for the property, but the "where" doesn't seem to actually generate and my query always returns 0 records.
The attribute is simple:
[AttributeUsage(AttributeTargets.Property, AllowMultiple=true)]
public class FilterAttribute: Attribute
{
public FilterType FilterType { get; set; } //enum{ Object, Database}
public string FilterPath { get; set; }
//var predicate = PredicateBuilder.False<Metadata>();
}
And this is my method that builds out the query:
public List<ETracker.Objects.Item> Search(Search SearchObject, int Page, int PageSize)
{
var predicate = PredicateBuilder.False<ETracker.Objects.Item>();
Type t = typeof(Search);
IEnumerable<PropertyInfo> pi = t.GetProperties();
string title = string.Empty;
foreach (var property in pi)
{
if (Attribute.IsDefined(property, typeof(FilterAttribute)))
{
var attrs = property.GetCustomAttributes(typeof(FilterAttribute),true);
var value = property.GetValue(SearchObject, null);
if (property.Name == "Title")
title = (string)value;
predicate.Or(a => GetPropertyVal(a, ((FilterAttribute)attrs[0]).FilterPath) == value);
}
}
var res = dataContext.GetAllItems().Take(1000)
.Where(a => SearchObject.Subcategories.Select(b => b.ID).ToArray().Contains(a.SubCategory.ID))
.Where(predicate);
return res.ToList();
}
The SearchObject is quite simple:
public class Search
{
public List<Item> Items { get; set; }
[Filter(FilterType = FilterType.Object, FilterPath = "Title")]
public string Title { get; set; }
...
}
Any suggestions will be greatly appreciated. I may well be going way the wrong direction and will take no offense if someone has a better alternative (or at least one that works)
You're not assigning your predicate anywhere. Change the line to this:
predicate = predicate.Or(a => GetPropertyVal(a, ((FilterAttribute)attrs[0]).FilterPath) == value);

Resources