I'm building a pure lucene-based implementation for legacy system that should locally (in-process) run arbitrary (more or less) queries. I receive queries in textual form, like term1:A OR term2:B. My default operator for queries is AND. For compatibility reasons I'm using lucene 5.5. I'm basing the solution on MemoryIndex single-document implementation.
My main problem is with how lucene deals with NOT clause: in query of sort
termA:(NOT 0) term2:(xxx) lucene would produce the following query:
+(-termA:0) +term2:xxx. While logically pretty sound, in practice this query shall always produce nothing since the negate-only clause always returns no_docs response and AND with it shall always produce empty result. The only workaround that I found (here on SOF) for this is to inject a MatchAllDocs clause together with the negative clause, like termA:(*:* NOT 0), but this is an error prone approach since it involves intricate clause parsing - the MatchAllDocs should only be injected if there are no positive clauses inside the parenthesis.
I'm looking for a more robust/generic approach to this problem, preferable some library that can parse/handle it for me, maybe something in contrib or similar.
Related
I'm looking for confirmation/clarification with these LINQ expressions:
var context = new SomeCustomDbContext()
// LINQ to Entities?
var items = context.CustomItems.OrderBy(i => i.Property).ToList();
// LINQ to Objects?
var items2 = context.CustomItems.ToList().OrderBy(i => i.Property);
Am I correct in thinking the first method is LINQ to Entities where EF builds a more specific SQL statement to pass on, putting the ordering effort on on the database?
Is the second method LINQ to Objects where LINQ drags the whole collection into memory (the ToList() enumeration?) before ordering thus leaving the burden on the server side (the web server in this case)?
If this is the case, I can quickly see situations where L2E would be advantageous (ex. filtering/trimming collections before pulling them into memory).
But are there any other details/trade-offs I should be aware of, or times when "method 2" might be advantageous over the first method?
UPDATE:
Let's say we are not using EntityFramework, this is still true so long as the underlying repository/data source implements IQueryable<T> right? And if it doesn't both these statements result in LINQ to Objects operations in memory?
Yes.
Yes.
Yes.
You are correct that calling ToList() forces linq-to-entities to evaluate and return the results as a list. As you suspect, this can have huge performance implications.
There are cases where linq-to-entities cannot figure out how to parse what looks like a perfectly simple query (like Where(x => SomeFunction(x))). In these cases you often have no choice but to call ToList() and operate on the collection in memory.
In response to your update:
ToList() always forces everything ahead of it to evaluate immediately, as opposed to deferred execution. Take this example:
someEnumerable.Take(10).ToList();
vs
someEnumerable.ToList().Take(10);
In the second example, any deferred work on someEnumerable must be executed before taking the first 10 elements. If someEnumerable is doing something labor intensive (like reading files from the disk using Directory.EnumerateFiles()), this could have very real performance implications.
Am I correct in thinking the first method is LINQ to Entities where EF builds a more specific SQL statement to pass on, putting the ordering effort on on the database?
Yes
Is the second method LINQ to Objects where LINQ drags the whole collection into memory ... before ordering thus leaving the burden on the server side ...?
Yes
But are there any other details/trade-offs I should be aware of, or times when "method 2" might be advantageous over the first method?
There will be many times where Method 1 is not possible - usually when you have a complex filter or sort order that can't be directly translated to SQL (or more appropriately where EF does not support a direct SQL translation). Also since you can't transmit lazy-loaded IQueryables over-the-wire, any time you have to serialize a result you're going to have to materialize it first with ToList() or something similar.
The other thing to be aware of is that IQueryable makes no guarantees on either (a) the semantic reasoning of the underlying provider, or (b) how much of the set of IQueryable methods are implemented by the provider.
For example: -
EF does not support Last().
Nor does it support time-part comparisons of DateTimes into valid T-SQL.
It doesn't support FirstOrDefault() in subqueries.
In such circumstances you need to bring data back to the client and then perform further evaluation client-side.
You also need to have an understanding of "how" it parses the LINQ pipeline to generate (in the case of EF) T-SQL. So you sometimes have to think carefully about how you construct your LINQ queries in order to generate effective T-SQL.
Having said all that, IQueryable<> is an extremely powerful tool in the .NET framework and well worth getting more familiar with.
I have not found a clear comparison of what is supported with the NHibernate 3.0 LINQ Provider compared to using the QueryOver syntax. From the surface, it seems like two large efforts into two very similar things.
What are the key trade offs to using each?
LINQ and QueryOver are completely different query methods, which are added to the ones that existed in NHibernate 2 (Criteria, HQL, SQL)
QueryOver is meant as a strongly-typed version of Criteria, and supports mostly the same constructs, which are NHibernate-specific.
LINQ is a "standard" query method, which means the client code can work on IQueryable without explicit references to NHibernate. It supports a different set of constructs; it would be hard to say if there are more or less than with QueryOver.
My suggestion is to learn all the supported query methods, as each use case is different and some work better with one, some work better with other.
I have used both NH-Linq-providers (the old NHContrib for Version 2.1, and also the new for NH3.0) and also used QueryOver. With all the experience made during development of quite complex data-driven applications, I would strongly suggest NOT to use the existing linq-provider with nHibernate if you plan to go behind just basic CRUD-operations!
The current implementation (linq) sometimes produces really unreadable and also unefficient SQL. Especially joining some tables quickly becomes a nightmare if you want to optimize database-performance.
Despite all these drawbacks, I did never encounter wrong queries.
So if you don't care about performance and are already familiar with LINQ, then go for NH-Linq. Otherwise QueryOver is your realiable and typesafe friend.
LINQ to NHibernate (as of version 3.0) does not support the .HasValue property on Nullable types. One must compare to null in queries.
I started to use NH-Linq, because i was already done with LinqToSql and Entity Framework. But, for more complex queries, i have always finished with QueryOver. Reasons:
It's happen that query with NH-Linq doesn't work as expected. I can't remember exactly, but it doesn't work correct with some complex queries. Seems that is too young. And as dlang stated in previous answer, it's produce unefficient SQL.
When you learn QueryOver, it's easy to call functions, do projections, subqueries, seems to me more easy then with NH-Linq.
Good thing for NH-Linq - it can be extended, like Fabio Maulo explained here. But, similar is quite possible with QueryOver, but not so fancy as with NH-Linq :)
I'm looking for a query parser and optimizer that can work with OSGi filter conditions, which are based on LDAP query syntax. Preferably something implemented in Java, but a different implementation (even web based) would be good too. My main use case is to feed it queries that maybe can be simplified.
I'm looking for an application to display what a linq expression would do, in particular regarding the usage of multiple access to the same list in a query.
Better yet, the tool would tell if the linq query is good.
I used the expression tree visualizer in the past to at least help decode what is inside of an expression tree. It aids in figureing out the parts of the tree and how gives each part is related.
Well, to begin with, I could easily foresee a tool that would pick a query apart and detect that the Where-clause is the standard runtime implementation, and thus not examine that method, but "know" what the execution plan for that method would be, and could thus piece together a plan for the whole query.
Right up until the point where you introduce a custom Linq provider, where the only way to figure out what it will be doing would be to read the code.
So I daresay there is no such tool, and making one would be very hard.
Would be fun to try though, at least for standard classes, would be a handy debugging visualizer for Visual Studio.
What about making the tool yourself?! ;)
Take a look at Expression trees, I believe they might be useful
I was thinking about making something like Linq for Lua, and I have a general idea how Linq works, but was wondering if there was a good article or if someone could explain how C# makes Linq possible
Note: I mean behind the scenes, like how it generates code bindings and all that, not end user syntax.
It's hard to answer the question because LINQ is so many different things. For instance, sticking to C#, the following things are involved:
Query expressions are "pre-processed" into "C# without query expressions" which is then compiled normally. The query expression part of the spec is really short - it's basically a mechanical translation which doesn't assume anything about the real meaning of the query, beyond "order by is translated into OrderBy/ThenBy/etc".
Delegates are used to represent arbitrary actions with a particular signature, as executable code.
Expression trees are used to represent the same thing, but as data (which can be examined and translated into a different form, e.g. SQL)
Lambda expressions are used to convert source code into either delegates or expression trees.
Extension methods are used by most LINQ providers to chain together static method calls. This allows a simple interface (e.g. IEnumerable<T>) to effectively gain a lot more power.
Anonymous types are used for projections - where you have some disparate collection of data, and you want bits of each of the aspects of that data, an anonymous type allows you to gather them together.
Implicitly typed local variables (var) are used primarily when working with anonymous types, to maintain a statically typed language where you may not be able to "speak" the name of the type explicitly.
Iterator blocks are usually used to implement in-process querying, e.g. for LINQ to Objects.
Type inference is used to make the whole thing a lot smoother - there are a lot of generic methods in LINQ, and without type inference it would be really painful.
Code generation is used to turn a model (e.g. DBML) into code
Partial types are used to provide extensibility to generated code
Attributes are used to provide metadata to LINQ providers
Obviously a lot of these aren't only used by LINQ, but different LINQ technologies will depend on them.
If you can give more indication of what aspects you're interested in, we may be able to provide more detail.
If you're interested in effectively implementing LINQ to Objects, you might be interested in a talk I gave at DDD in Reading a couple of weeks ago - basically implementing as much of LINQ to Objects as possible in an hour. We were far from complete by the end of it, but it should give a pretty good idea of the kind of thing you need to do (and buffering/streaming, iterator blocks, query expression translation etc). The videos aren't up yet (and I haven't put the code up for download yet) but if you're interested, drop me a mail at skeet#pobox.com and I'll let you know when they're up. (I'll probably blog about it too.)
Mono (partially?) implements LINQ, and is opensource. Maybe you could look into their implementation?
Read this article:
Learn how to create custom LINQ providers
Perhaps my LINQ for R6RS Scheme will provide some insights.
It is 100% semantically, and almost 100% syntactically the same as LINQ, with the noted exception of additional sort parameters using 'then' instead of ','.
Some rules/assumptions:
Only dealing with lists, no query providers.
Not lazy, but eager comprehension.
No static types, as Scheme does not use them.
My implementation depends on a few core procedures:
map - used for 'Select'
filter - used for 'Where'
flatten - used for 'SelectMany'
sort - a multi-key sorting procedure
groupby - for grouping constructs
The rest of the structure is all built up using a macro.
Bindings are stored in a list that is tagged with bound identifiers to ensure hygiene. The binding are extracted and rebound locally where ever an expression occurs.
I did track the progress on my blog, that may provide some insight to possible issues.
For design ideas, take a look at c omega, the research project that birthed Linq. Linq is a more pragmatic or watered down version of c omega, depending on your perspective.
Matt Warren's blog has all the answers (and a sample IQueryable provider implementation to give you a headstart):
http://blogs.msdn.com/mattwar/