The documentation for LINQ to SQL translation for Cosmos DB states:
User-Defined Function Extension function: Supports translation from the stub method UserDefinedFunctionProvider.Invoke to the corresponding user-defined function.
However, this function is not publicly accessible in .NET SDK v3 (though it is in v2). So what's the workaround until the bug has been fixed?
There are several workarounds:
Don't use LINQ.
Use LINQ but without calling UDFs, then get the query string using query.ToQueryDefinition.QueryText and manipulate it to insert UDF calls at the places you need them, and then evaluate the SQL string (yuck!).
Hack it. First add a dummy method:
public static object Invoke(string udfName, object[] arguments) { return null; }
Use this method in your lambda expressions whenever you would otherwise use UserDefinedFunctionProvider.Invoke. Implement an ExpressionVisitor that replaces all occurrences of MethodCallExpression mce where mce.Method matches the dummy method by
Expression.Call(null, udfMethod, mce.Arguments[0], mce.Arguments[1])
where udfMethod is
MethodInfo udfMethod = typeof(CosmosClient).Assembly
.GetType("Microsoft.Azure.Cosmos.Linq.UserDefinedFunctionProvider", throwOnError: true)
.GetMethod("Invoke", BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static);
Use the resulting expression in your Where, Select etc.
(Add a unit test to check when the bug has been fixed and the hack can be removed.)
I am using the Microsoft.Azure.Cosmos 3.22.1 SDK for my example with the UDF (user defined functions) and it worked fine.
I used the example from microsoft.
It is of course also possible to use the UDF in combination with where clauses.
CosmosClient client = new(endpoint, key);
IQueryable<Product> queryable = client
.GetContainer("database", "products")
.GetItemLinqQueryable<Product>()
.Select(b => new Product{ id = b.id, price = b.price, priceWithTax = CosmosLinq.InvokeUserDefinedFunction("tax", b.price).ToString() });
var productIterator = queryable.ToFeedIterator<Product>();
while (productIterator.HasMoreResults)
{
var responseMessage = await productIterator.ReadNextAsync();
Console.WriteLine(Newtonsoft.Json.JsonConvert.SerializeObject(responseMessage));
}
Related
I am using the repository pattern with Entity Framework as described in this article: repository pattern with Entity Framework
In the part where the GenericRepository is described (Generic Repository) there is a method which is used to get entities from the database set called Get. It has an orderBy but no groupBy. I am wondering how one might implement a groupBy in the same manner as the orderBy so that you can specify which field to group by dynamically on the entity.
What I have come up with is this:
Func<IQueryable<TEntity>, IGrouping<string, TEntity>> groupBy = null
and then in the method code it should be used something like this:
if(groupBy != null)
{
query = groupBy(query).ToList();
}
But this is not compiling since the IGrouping is not queryable. Does someone know how to point me in the right direction or has a solution to this?
Edit: The reason for doing this instead of using groupby on the returned list is for performance reasons. I want the groupby to be sent as an sql statement to the database and resolved there.
Grouping has no sense without projection. So you have to define new method which returns IEnumerable with new type.
I have added sample of such method. Also removed includeProperties because EF Core ignores Includes during grouping.
Usage sample:
_orderRepostory
.GetGrouped(e => e.UserId, g => new { UserId = g.Key, Count = g.Count()});
And implementation:
public class GenericRepository<TEntity> where TEntity : class
{
... // other code
public virtual IEnumerable<TResult> GetGrouped<TKey, TResult>(
Expression<Func<TEntity, TKey>> groupingKey,
Expression<Func<IGrouping<TKey, TEntity>, TResult>> resultSelector,
Expression<Func<TEntity, bool>>? filter = null)
{
var query = dbSet.AsQueryable();
if (filter != null)
{
query = query.Where(filter);
}
return query.GroupBy(groupingKey).Select(resultSelector);
}
}
How to use string extension method in linq query:
public NewsType GetNewsType(string name)
{
var newsType = db.NewsTypes.FirstOrDefault(x => x.Name.ToFriendlyUrl() ==
name.ToFriendlyUrl());
return newsType;
}
Above query x.Name.ToFriendlyUrl() is not allowed at the minute. Is anyone know how to achieve with it.
Extension methods are indeed allowed in LINQ queries, moreover the LINQ methods themselves are implemented as extension methods.
It's quite another issue however, to use extension methods (or most other methods) in LINQ-to-SQL or LINQ-to-Entities queries. Those queries are not actually run in the C# code, but they are treated like expressions that are translated to SQL. I.e.
db.News.Where(x => x.Published).Select(x => x.Name)
is translated to the SQL Statement
Select Name
From News
Where Published = 1
and it's results are returned to the C# code.
Since there is not way to transfer the ToFriendlyUrl() method to SQL, your code throws an error.
You have basically, two solutions/workarounds. One is to transform the call to a form could be translated into SQL, e.g. if the ToFriendlyUrl() method was just:
public static string ToFriendlyURL(this string value)
{
return value.ToLower();
}
you can inline that code in the LINQ call, and that would work. If however, the methods is more complex, than your only solution is to just fetch the data from the base and then process it on the C# side:
var newsTypeQuery = db.NewsTypes.Where(x => // other conditions, if any);
var newsTypes = newsTypes.ToList(); //forces execution of the query
// the result is now a C# list
var newsType = newsTypes.FirstOrDefault(x =>
x.Name.ToFriendlyUrl() == name.ToFriendlyUrl());
Assuming the NewsTypes is an IQueryable this is a result of Entity Framework not being able to convert you extension method into SQL (how should it?). Unless you can rewrite your predicate into something that Entity Framework can translate into SQL you will have to perform the query client side:
public NewsType GetNewsType(string name)
{
var newsType = db.NewsTypes.AsEnumerable().FirstOrDefault(x => x.Name.ToFriendlyUrl() == name.ToFriendlyUrl());
return newsType;
}
Notice how AsEnumerable() has been added before FirstOrDefault. Unfortunately this may pull all the rows returned by NewsTypes from the server to client and thus may be quite costly.
This
var newsType = db.NewsTypes.FirstOrDefault(
x => x.Name.ToFriendlyUrl() == name.ToFriendlyUrl());
can't be done in Entity Framework. ToFriendlyUrl is an extension method. It's something that is in the "client" computer. The query will be executed on the SQL server. The SQL server doesn't have a ToFriendlyUrl function.
The "standard" solution is to save in a second column named FriendlyName a precalculated version of the ToFriendlyUrl(), so your query becomes:
var friendlyName = name.ToFriendlyUrl();
var newsType = db.NewsTypes.FirstOrDefault(
x => x.FriendlyName == friendlyName);
Instead try like this
public NewsType GetNewsType(string name)
{
var newsType = db.NewsTypes.FirstOrDefault(x => x.Name == name).ToFriendlyUrl();
return newsType;
}
I am trying to send a Linq query as a string to a method to be used in a where clause. Since IEnumerable wouldn't work for this, I have converted my IEnumerable to IQueryable and still it throws error. The following is the code:
public static void FilterData(string Query)
{
if((List<MemberMaintenanceData>)HttpContext.Current.Session["Allmembers"] != null)
{
//Get the IEnumerable object colection from session
var data = (List<MemberMaintenanceData>) HttpContext.Current.Session["Allmembers"];
//Convert it to IQueryable
IQueryable<MemberMaintenanceData> queryData = data.AsQueryable();
//This line doesn't compile!!
queryData = queryData.Where(Query);
HttpContext.Current.Session["Allmembers"] = queryData.AsEnumerable().ToList();
}
}
I intended passing "a => a.AccountId == 1000" as Query
There is a free (and open source) library, provided by Microsoft for parsing strings into Lambda expressions that can then be used in Linq queries. It also contains versions of the standard query operators such as Where() that take a string parameter. You can find it described in Scott Guthries blog post on Dynamic Linq.
For example, you can do queries like this (adapted from a snippet from the Scott guthrie link)
// imagine these have come from a drop down box or some other user input...
string thingToSelectBy = "City";
string citySelectedByUser = "London";
int minNumberOfOrders = 10;
string whereClause = String.Format("{0} = #0 and Orders.Count >= #1", thingToSelectBy);
var query = db.Customers
.Where(whereClause, citySelectedByUser, minNumberOfOrders)
.OrderBy("CompanyName")
.Select("new(CompanyName as Name, Phone");
The Where clause in thisw code snippet shows how you create a where clause using a parameterised string and then dynamically inject values for the parameters at run time, for example, based on user input. This works for parameters of any type.
In your example, the where clause would be
whereClause = "AccountId = 1000";
So in effect you would be doing something like
var newFilteredQueryData = queryData.Where("AccountId = 1000");
That link also contains the location where you can download the source code and a comprehensive document describing the dynamic query API and expression language.
Given a class such as:
public class foo
{
public int AccountID {get;set;}
}
You should be able to do something like this:
Expression<Func<foo, bool>> filter = f => f.AccountID == 1000;
And then pass that as your query. If it is really needed as a string you can do this:
filter.ToString();
//By Using this library
using System.Linq.Dynamic.Core;
InventoryList = Repository.GetAll(); // IQueryable
string filterString = "UnitPrice > 10 And Qty>100 OR Description.Contains("Dairy")";
var filteredGenericList = InventoryList.Where(filterString);
I am trying to query a Azure Table Storage. For that I use the following two methods:
TableServiceContext:
public IQueryable<T> QueryEntities<T>(string tableName) where T : TableServiceEntity
{
this.ResolveType = (unused) => typeof(T);
return this.CreateQuery<T>(tableName);
}
Code that uses the method above:
CloudStorageAccount account = AzureConnector.GetCloudStorageAccount(AppSettingsVariables.TableStorageConnection);
AzureTableStorageContext context = new AzureTableStorageContext(account.TableEndpoint.ToString(), account.Credentials);
// Checks if the setting already exists in the Azure table storage. Returns null if not exists.
var existsQuery = from e in context.QueryEntities<ServiceSettingEntity>(TableName)
where e.ServiceName.Equals(this.ServiceName) && e.SettingName.Equals(settingName)
select e;
ServiceSettingEntity existingSettginEntity = existsQuery.FirstOrDefault();
The LINQ query above generates the following request url:
http://127.0.0.1:10002/devstoreaccount1/PublicSpaceNotificationSettingsTable()?$filter=(ServiceName eq 'PublicSpaceNotification') and (SettingName eq 'expectiss')
The code in the class generates the following MissingMethodException:
I have looked at the supported LINQ Queries for the Table API;
Looked at several working stackoverflow solutions;
Tried IgnoreResourceNotFoundException on the TableServiceContext (usercomments of QueryOperators);
Tried to convert the linq query with ToList() before calling first or default (usercomments of QueryOperators).
but I can't get this to work.
Make sure you have parameterless constructor for the class "ServerSettingEntity". The ‘DTO’ that inherits TableServiceEntity needs a constructor with no parameters.
I have an entity that I'd like to compare with a subset and determine to select all except the subset.
So, my query looks like this:
Products.Except(ProductsToRemove(), new ProductComparer())
The ProductsToRemove() method returns a List<Product> after it performs a few tasks. So in it's simplest form it's the above.
The ProductComparer() class looks like this:
public class ProductComparer : IEqualityComparer<Product>
{
public bool Equals(Product a, Product b)
{
if (ReferenceEquals(a, b)) return true;
if (ReferenceEquals(a, null) || ReferenceEquals(b, null))
return false;
return a.Id == b.Id;
}
public int GetHashCode(Product product)
{
if (ReferenceEquals(product, null)) return 0;
var hashProductId = product.Id.GetHashCode();
return hashProductId;
}
}
However, I continually receive the following exception:
LINQ to Entities does not recognize
the method
'System.Linq.IQueryable1[UnitedOne.Data.Sql.Product]
Except[Product](System.Linq.IQueryable1[UnitedOne.Data.Sql.Product],
System.Collections.Generic.IEnumerable1[UnitedOne.Data.Sql.Product],
System.Collections.Generic.IEqualityComparer1[UnitedOne.Data.Sql.Product])'
method, and this method cannot be
translated into a store expression.
Linq to Entities isn't actually executing your query, it is interpreting your code, converting it to TSQL, then executing that on the server.
Under the covers, it is coded with the knowledge of how operators and common functions operate and how those relate to TSQL. The problem is that the developers of L2E have no idea how exactly you are implementing IEqualityComparer. Therefore they cannot figure out that when you say Class A == Class B you mean (for example) "Where Person.FirstName == FirstName AND Person.LastName == LastName".
So, when the L2E interpreter hits a method it doesn't recognize, it throws this exception.
There are two ways you can work around this. First, develop a Where() that satisfies your equality requirements but that doesn't rely on any custom method. In other words, test for equality of properties of the instance rather than an Equals method defined on the class.
Second, you can trigger the execution of the query and then do your comparisons in memory. For instance:
var notThisItem = new Item{Id = "HurrDurr"};
var items = Db.Items.ToArray(); // Sql query executed here
var except = items.Except(notThisItem); // performed in memory
Obviously this will bring much more data across the wire and be more memory intensive. The first option is usually the best.
You're trying to convert the Except call with your custom IEqualityComparer into Entity SQL.
Obviously, your class cannot be converted into SQL.
You need to write Products.AsEnumerable().Except(ProductsToRemove(), new ProductComparer()) to force it to execute on the client. Note that this will download all of the products from the server.
By the way, your ProductComparer class should be a singleton, like this:
public class ProductComparer : IEqualityComparer<Product> {
private ProductComparer() { }
public static ProductComparer Instance = new ProductComparer();
...
}
The IEqualityComparer<T> can only be executed locally, it can't be translated to a SQL command, hence the error