Expression<Func<T, bool>> from a F# func - linq

in linq, .Where takes a Expression> predicate, which I can write in F# as
<# fun item:'a -> condition #> // Expr<'a -> bool>
I'm using FSharp.Powerpack to build the expression from a quotation, but what it gives me is a MethodCallExpression. Looking deep, the powerpack code builds the lambda correctly, but wraps it in a Convert call (why is that?). I wonder if casting the argument to the method call (a lambda) would finally give me the Expression> I need.
So the question is why the Convert call, and how to actually get the lambda with the Func signature.

I can't remember off the top of my head where I found this bit of code, but this is what I use to convert an Expr<'a -> 'b> to Expression<Func<'a, 'b>>. Hopefully this will solve your problem.
open System
open System.Linq.Expressions
open Microsoft.FSharp.Quotations
open Microsoft.FSharp.Linq.QuotationEvaluation
let toLinq (expr : Expr<'a -> 'b>) =
let linq = expr.ToLinqExpression()
let call = linq :?> MethodCallExpression
let lambda = call.Arguments.[0] :?> LambdaExpression
Expression.Lambda<Func<'a, 'b>>(lambda.Body, lambda.Parameters)

One way you can now do this is to take advantage of the fact that F# will perform this conversion automatically when invoking methods on .NET types that expect a Expression<Func<...>>.
I'm not entirely sure when this got added to the language, but certainly with F# 4, you don't need to explicitly convert F# expressions into LINQ ones. If the reason you wanted to do this in the first place was to be able to use IQueryable LINQ APIs (or other expression-based .NET APIs) then it now just works with no effort, e.g.:
someEfDataContext.MyEntities.Single(fun e -> e.Id = 42)
just works. Even though this looks like an ordinary lambda (we've not used F#'s expression syntax), this compiles to code that produces an F# expression object, and then passes that to LeafExpressionConverter‌​.QuotationToExpressi‌on to turn it into a LINQ expression object.
But sometimes you'll want to get hold of the LINQ-style expression object directly in F#. (E.g., sometimes it's useful to write an F# function that produces an expression that you'll use in multiple queries.) In that case you can write a helper like this:
type FunAs() =
static member LinqExpression<'T, 'TResult>(e: Expression<Func<'T, 'TResult>>) = e
This looks like it does nothing - it just returns its argument. However, because FunAs is a .NET type, F# will automatically compile any call site that invokes this with a fun expression into code that generates a suitable LINQ query expression. E.g.:
let linqExpr = FunAs.LinqExpression(fun (e:MyEntity) -> e.Id = 42)
Here, linqExpr will be of type Expression<Func<MyEntity, bool>>.
The key to this is that this method is a member of a .NET Type. If you try the exact same thing with an ordinary F# function:
let funAsLinqExpression<'T, 'TResult>(e: Expression<Func<'T, 'TResult>>) = e
which seems like it should mean exactly the same thing as FunAs.LinqExpression, you'll find that you can't call it in the same way. E.g., if you try this:
let linqExpr = funAsLinqExpression(fun (e:MyEntity) -> e.Id = 42)
You'll get a (slightly unhelpful) error: 'This function takes too many arguments, or is used in a context where a function is not expected`.
By making this function a member of a .NET type, we can take advantage of F#'s helpful "You seem to be invoking a .NET API that expects a LINQ-style expression, let me take care of that for you" feature.
(It's possible that there's some more explicit way of asking the LINQ compiler to perform this same trick for you without bringing a .NET type into the picture, but I've not found it.)

Related

Is F# Constructed Type syntax special?

I was curious about F#'s "constructed type" syntax. It's documented here.
type-argument generic-type-name
or
generic-type-name
With the following examples:
int option
string list
int ref
option<int>
list<string>
ref<int>
Dictionary<int, string>
I was curious if there's anything special about the "backwards" syntax, with the parameter before the type, or if it's just sugar for generic types with one parameter. The following is valid:
type 'a MyOption = // MyOption<'a> also works
| MySome of 'a
| MyNone
But I could not get it to work with multiple type parameters. Why do F# developers prefer this syntax for types with one parameter? Is it possible or desirable to make it work with two?
The backwards syntax is a legacy from OCaml. Personally, I never use it. If you really want to, you can make it work with multiple type arguments like this:
type MyMap = (int, string) Map
However, this generates a pointed warning (that might soon become an error):
This construct is for ML compatibility. The syntax '(typ,...,typ) ident' is not used in F# code. Consider using 'ident<typ,...,typ>' instead. You can disable this warning by using '--mlcompatibility' or '--nowarn:62'.
Bottom line, I would recommend always using .NET syntax instead: MyOption<'a> instead of 'a MyOption.
Why do F# developers prefer this syntax for types with one parameter?
Not all of us do. I love F# and am in awe of it, but find the OCaml style distracting.
It gets especially confusing when the two styles are mixed - compare the readability of Async<Result<int,string list>> list with that of List<Async<Result<int,List<string>>>>.
Here is a thread with some arguments from both sides from fslang suggestions, which I think led to the deprecation of OCaml-style for everything but list, option and a few others.
I find it regrettable that the OCaml style is specified as the preferred option (for these types) in the various style guides, and used throughout the core libraries, while there is such a strong drive to make the language more accessible to newcomers. It definitely adds to the learning curve, as documented in this question,
and here,
here,
here,
here,
here.
Is it possible or desirable to make [OCaml style naming] work with two [type parameters]?
I think a better question is: "Is it possible to only use .NET style?".
Unfortunately the tooling shows types the way they are declared, and the core libraries consistently use OCaml style. I have asked Rider about always showing declarations .NET style in code vision, who referred me to FSharp compiler services. I have not (yet) investigated that avenue further.
In our own code we have taken to overriding the OCaml signatures of functions that ship with F# and other libraries as we come across them, for example:
[<AutoOpen>]
module NoCaml =
module List =
/// Returns a new collection containing only the elements of the collection for which the given predicate returns "true"
let filter = List.filter : ('a -> bool) -> List<'a> -> List<'a>
/// val groupBy : projection:('T -> 'Key) -> list:'T list -> ('Key * 'T list) list (requires equality and equality) 'T is 'a 'Key is 'b Applies a key-generating function to each element of a list and yields a list of unique keys. Each unique key contains a list of all elements that match to this key.
let groupBy = List.groupBy : ('a -> 'b) -> List<'a> -> List<'b * List<'a>>
// etc.
This solves the problem in almost all cases (some exceptions like list construction using [] remain, and need to be overridden at the point of declaration).
I'm not sure what influence this has on performance at runtime - hopefully the extra function calls are optimised away.

Why are parenthesis needed in F# using method chaining query expression where clause?

Attempting to write more queries in F# through LinqPad I ran into an interesting thing with the where clause.
let dc = new TypedDataContext()
let q = query {
for o in dc.OrderItem do
where (o.Description.Contains("spam"))
select o
}
q |> Dump
If I remove the parenthesis around o.Description.Contains("spam") I get the often seen error message below.
Successive arguments should be separated by spaces or tupled, and arguments involving function or method applications should be parenthesized (using external F# compiler)
When I see this error I normally realize that I'm not giving enough information in order for the compiler to understand that I'm attempting to chain off of the result of a prior method call or property get accessor invocation but in this case it is unclear to me. Additionally I'd be curious if there is a more idiomatic way of satisfying the conditions of the compiler without having to reach around the expression to add both open and closing parenthesis.
You'd see the same result if you did something like this:
let f x = 1
f o.Description.Contains("spam")
As the message implies, if you're using the result of a method call (Contains in this case) as an argument to a function, then the method call needs to be parenthesized. Even though where is a query operator rather than a true function, the same result applies.

Groovy 1.8 :: LINQ Applied

UPDATE 8/31/2011
Guillaume Laforge has nearly done it:
http://gaelyk.appspot.com/tutorial/app-engine-shortcuts#query
Looks like he's doing an AST transform to pull off the:
alias as Entity
bit. Cool stuff, Groovy 1.8 + AST transform = LINQ-esque queries on the JVM. GL's solution needs more work as far as I can see to pull off full query capabilities (like sub queries, join using(field) syntax and the like), but for his Gaelyk project apparently not necessary.
EDIT
As a workaround to achieving pure LINQ syntax, I have decided to def the aliases. Not a huge deal, and removes a major hurdle that would likely require complex AST transforms to pull off.
So, instead of:
from c as Composite
join t as Teams
...
I now define the aliases (note: need to cast to get auto complete on fields):
def(Teams t,Composite c,Schools s) = [Teams.new(),Composite.new(),Schools.new()]
and use map syntax for from, join, etc.
from c:Composite
join t:Teams
...
To solve issue #2 (see original below), add instance level getProperty methods to each pogo alias (whose scope is limited to ORM closure in which it is called, nice). We just return the string property name as we're building an sql statement.
[t,c,s].each{Object o-> o.metaClass.getProperty = { String k-> k } }
Making "good" progress ;-)
Now to figure out what to do about "=", that is tricky since set property is void. May have to go with eq, neq, gt, etc. but would really prefer the literal symbols, makes for closer-to-sql readability.
If interested, LINQ is doing quite a bit behind the scenes. Jon Skeet (praise his name) has a nice SO reply:
How LINQ works internally?
ORIGINAL
Have been checking out LINQ, highly impressed.
// LINQ example
var games =
from t in Teams
from g in t.Games
where g.gameID = 212
select new { g.gameDate,g.gameTime };
// Seeking Groovy Nirvana
latest { Integer teamID->
from c as Composite
join t as Teams
join s as Schools on ( schoolID = {
from Teams
where t.schoolID = s.schoolID } )
where t.teamID = "$teamID"
select c.location, c.gameType, s.schoolName
group c.gameID
order c.gameDate, c.gameTime
}
The proposed Groovy version compiles fine and if I def the aliases c,t,s with their corresponding POGO, I get strong typed IDE auocomplete on fields, nice. However, nowhere near LINQ, where there are no (visible) variable definitions other than the query itself, totally self contained and strongly typed, wow.
OK, so can it be done in Groovy? I think (hope) yes, but am hung up on 2 issues:
1) How to implicitly populate alias variable without def'ing? Currently I am overriding asType() on String so in "from c as Composite", c gets cast to Composite. Great, but the IDE "thinks" that in closure scope undefined c is a string and thus no autocomplete on POGO fields ;-(
2) Since #1 is not solved, I am def'ing the aliases as per above so I can get autocomplete. Fine, hacked (compared to LINQ), but does the trick. Problem here is that in "select c.location, c.gameType...", I'd like the fields to not be evaluated but simply return "c.location" to the ORM select method, and not null (which is its default value). getProperty() should work here, but I need it to only apply to pogo fields when called from ORM scope (e.g. orm field specific methods like select, order, group, etc.). Bit lost there, perhaps there's a way to annotate orm methods, or only invoke "special" pogo getProperty via orm method calls (which is the closure's delegate in above nirvana query).
Should point out that I am not looking to create a comprehensive LINQ for Groovy, but this one particular subset of LINQ I would love to see happen.
One of the biggest reasons for Guillaume to use an AST transform is because of the problem with "=". Even if you use == for the compare as normally done in Groovy, from the compareTo method that is called for it, you cannot make out a difference between ==, !=, <=, >=, <, >. There are two possible paths for this in later versions of Groovy that are in discussion. One is to use for each of those compares a different method, the other is to store a minimal AST which you can access at runtime. This goes in the direction of C# and is a quite powerful tool. The problem is more about how to do this efficiently.

How LINQ works internally?

I love using LINQ in .NET, but I want to know how that works internally?
It makes more sense to ask about a particular aspect of LINQ. It's a bit like asking "How Windows works" otherwise.
The key parts of LINQ are for me, from a C# perspective:
Expression trees. These are representations of code as data. For instance, an expression tree could represent the notion of "take a string parameter, call the Length property on it, and return the result". The fact that these exist as data rather than as compiled code means that LINQ providers such as LINQ to SQL can analyze them and convert them into SQL.
Lambda expressions. These are expressions like this:
x => x * 2
(int x, int y) => x * y
() => { Console.WriteLine("Block"); Console.WriteLine("Lambda"); }
Lambda expressions are converted either into delegates or expression trees.
Anonymous types. These are expressions like this:
new { X=10, Y=20 }
These are still statically typed, it's just the compiler generates an immutable type for you with properties X and Y. These are usually used with var which allows the type of a local variable to be inferred from its initialization expression.
Query expressions. These are expressions like this:
from person in people
where person.Age < 18
select person.Name
These are translated by the C# compiler into "normal" C# 3.0 (i.e. a form which doesn't use query expressions). Overload resolution etc is applied afterwards, which is absolutely key to being able to use the same query syntax with multiple data types, without the compiler having any knowledge of types such as Queryable. The above expression would be translated into:
people.Where(person => person.Age < 18)
.Select(person => person.Name)
Extension methods. These are static methods which can be used as if they were instance methods of the type of the first parameter. For example, an extension method like this:
public static int CountAsciiDigits(this string text)
{
return text.Count(letter => letter >= '0' && letter <= '9');
}
can then be used like this:
string foo = "123abc456";
int count = foo.CountAsciiDigits();
Note that the implementation of CountAsciiDigits uses another extension method, Enumerable.Count().
That's most of the relevant language aspects. Then there are the implementations of the standard query operators, in LINQ providers such as LINQ to Objects and LINQ to SQL etc. I have a presentation about how it's reasonably simple to implement LINQ to Objects - it's on the "Talks" page of the C# in Depth web site.
The way providers such as LINQ to SQL work is generally via the Queryable class. At their core, they translate expression trees into other query formats, and then construct appropriate objects with the results of executing those out-of-process queries.
Does that cover everything you were interested in? If there's anything in particular you still want to know about, just edit your question and I'll have a go.
LINQ is basically a combination of C# 3.0 discrete features of these:
local variable type inference
auto properties (not implemented in VB 9.0)
extension methods
lambda expressions
anonymous type initializers
query comprehension
For more information about the journey to get there (LINQ), see this video of Anders in LANGNET 2008:
http://download.microsoft.com/download/c/e/5/ce5434ca-4f54-42b1-81ea-7f5a72f3b1dd/1-01%20-%20CSharp3%20-%20Anders%20Hejlsberg.wmv
In simple a form, the compiler takes your code-query and converts it into a bunch of generic classes and calls. Underneath, in case of Linq2Sql, a dynamic SQL query gets constructed and executed using DbCommand, DbDataReader etc.
Say you have:
var q = from x in dc.mytable select x;
it gets converted into following code:
IQueryable<tbl_dir_office> q =
dc.mytable.Select<tbl_dir_office, tbl_dir_office>(
Expression.Lambda<Func<mytable, mytable>>(
exp = Expression.Parameter(typeof(mytable), "x"),
new ParameterExpression[] { exp }
)
);
Lots of generics, huge overhead.
Basically linq is a mixture of some language facilities (compiler) and some framework extensions. So when you write linq queries, they get executed using appropriate interfaces such as IQuerable. Also note that the runtime has no role in linq.
But it is difficult to do justice to linq in a short answer. I recommend you read some book to get yourself in it. I am not sure about the book that tells you internals of Linq but Linq in Action gives a good handson about it.

How does coding with LINQ work? What happens behind the scenes?

For example:
m_lottTorqueTools = (From t In m_lottTorqueTools _
Where Not t.SlotNumber = toolTuple.SlotNumber _
And Not t.StationIndex = toolTuple.StationIndex).ToList
What algorithm occurs here? Is there a nested for loop going on in the background? Does it construct a hash table for these fields? I'm curious.
Query expressions are translated into extension method calls, usually. (They don't have to be, but 99.9% of queries use IEnumerable<T> or IQueryable<T>.)
The exact algorithm of what that method does varies from method to method. Your sample query wouldn't use any hash tables, but joins or grouping operations do, for example.
The simple Where call translates to something like this in C# (using iterator blocks, which aren't available in VB at the moment as far as I'm aware):
public static IEnumerable<T> Where(this IEnumerable<T> source,
Func<T, bool> predicate)
{
// Argument checking omitted
foreach (T element in source)
{
if (predicate(element))
{
yield return element;
}
}
}
The predicate is provided as a delegate (or an expression tree if you're using IQueryable<T>) and is called on each item in the sequence. The results are streamed and execution is deferred - in other words, nothing happens until you start asking for items from the result, and even then it only does as much as it needs to in order to provide the next result. Some operators aren't deferred (basically the ones which return a single value instead of a sequence) and some buffer the input (e.g. Reverse has to read to the end of the sequence before it can return any results, because the last result it reads is the first one it has to yield).
It's beyond the scope of a single answer to give details of every single LINQ operator I'm afraid, but if you have questions about specific ones I'm sure we can oblige.
I should add that if you're using LINQ to SQL or another provider that's based on IQueryable<T>, things are rather different. The Queryable class builds up the query (with the help of the provider, which implements IQueryable<T> to start with) and then the query is generally translated into a more appropriate form (e.g. SQL) by the provider. The exact details (including buffering, streaming etc) will entirely depend on the provider.
LINQ in general has a lot going on behind the scenes. With any query, it is first translated into an expression tree using an IQueryableProvider and from there the query is typically compiled into CIL code and a delegate is generated pointing to this function, which you are essentially using whenever you call the query. That's an extremely simplified overview - if you want to read a great article on the subject, I recommend you look at How LINQ Works - Creating Queries. Jon Skeet also posted a good answer on this site to this question.
What happens not only depends on the methods, but also depends on the LINQ provider in use. In the case where an IQueryable<T> is being returned, it is the LINQ provider that interprets the expression tree and processes it however it likes.

Resources