I need to select an unknown number of properties from a table.
For example, if I have a table with columns : c1, c2, c3, cn...
I have to generate a specific query that can get any of these columns, I don´t know which ones neither how many.
So I've tried to build a string that comes like:
selector = "e => new {
s3 = e.s3,
s2 = e.s2,
s1 = e.s1}"
and then
result = myQueriable.Select(selector);
The problem is that using System.Linq.Dynamic.Core
I get the exception: "Exception: No property or field 's3' exists in type 'MyTable'"
It works fine if I use this like:
myQueriable.Select(e => new {s3 = e.s3, s2 = e.s2,s1 = e.s1})
But I really need to make it dynamic because I don't know which properties and how many properties I'll need to select.
Due to the giant amount of data in this table is very important that the selection is translated and run in the db and not in client side.
Any suggestions?
Thanks
Related
I am new to Pig so bear with me. I have two datasources that have the same schema: a map of attributes. I know that some attributes will have a single identifiable overlapping attribute. For example
Record A:
{"Name":{"First":"Foo", "Last":"Bar"}, "FavoriteFoods":{["Oranges", "Pizza"]}}
Record B:
{"Name":{"First":"Foo", "Last":"Bar"}, "FavoriteFoods":{["Buffalo Wings"]}}
I want to merge the records on Name such that:
Merged:
{"Name":{"First":"Foo", "Last":"Bar"}, "FavoriteFoods":{["Oranges", "Pizza", "Buffalo Wings"]}}
UNION, UNION ONSCHEMA,and JOIN don't operate in this way. Is there a method available to do this within Pig or will it have to happen within a UDF?
Something like:
A = LOAD 'fileA.json' USING JsonLoader AS infoMap:map[];
B = LOAD 'fileB.json' USING JsonLoader AS infoMap:map[];
merged = MERGE_ON infoMap#Name, A, B;
Pig by itself is very dumb when it comes to even slightly complex data translation. I feel you will need two kinds of UDFs to achieve your task. The first UDF will need to accept a map and create a unique string representation of it. It could be like a hashed string representation of the map (lets call it getHashFromMap()). This string will be used to join the two relations. The second UDF would accept two maps and return a merged map (lets call it mergeMaps()). Your script will then look as follows:
A = LOAD 'fileA.json' USING JsonLoader AS infoMapA:map[];
B = LOAD 'fileB.json' USING JsonLoader AS infoMapB:map[];
A2 = FOREACH A GENERATE *, getHashFromMap(infoMapA#'Name') AS joinKey;
B2 = FOREACH B GENERATE *, getHashFromMap(infoMapB#'Name') AS joinKey;
AB = JOIN A2 BY joinKey, B2 BY joinKey;
merged = FOREACH AB GENERATE *, mergeMaps(infoMapA, infoMapB) AS mergedMap;
Here I assume that the attrbute you want to merge on is a map. If that can vary, you first UDF will need to become more generic. Its main purpose would be to get a unique string representation of the the attribute so that the datasets can be joined on that.
Here is my scenario. I have a Document class. The document is associated with a DocumentClasses table through a many to many relationship, so a document can have one or more classes. When running a search, the user can choose to filter documents by class. So, I need to be able to append a where clause to my query if the user chooses to select any classes. The logic is that if the document is assigned to any of the classes in the classes the user selected, the document should be returned. So the basic query needed in pseudo code. So basically, if any number in list A belongs to list B, then return the record.
I have tried this (RestrictByClasses is just a List(Of Integer)):
query = query.where(Function(resultItem) RestrictByClasses.Contains(resultItem.DocumentClassIds.Any())
But I get the following exception:
The nested query is not supported. Operation1='Case' Operation2='Collect'
Is there any way to get linq to filter records out like this?
Thanks!
UPDATE:
After doing a little more debugging, I think that it is more to do with how I am projecting onto the object in order to load it with values that can be used to filter. Here is how I am doing the projection:
Dim query = From document In dbContext.Documents
Select New FeeAndReceptionReportIntermediateItem With
{
.BookTypeId = If(restrictByBookTypes AndAlso document.DocumentInstruments.Any(), document.DocumentInstruments.FirstOrDefault().Instrument.BookTypeId, Nothing),
.CustomerId = document.CustomerId,
.DocumentClassIds = If(restrictByDocumentClasses, document.DocumentClasses.Select(Function(group) group.ClassId), Nothing),
.DocumentId = document.DocumentId,
.DocumentNumber = document.DocumentNumber,
.DepartmentId = document.DepartmentId,
.InstrumentGroupIds = If(restrictByInstrumentGroup, document.DocumentInstruments.FirstOrDefault().Instrument.InstrumentGroups.Select(Function(group) group.InstrumentGroupId), Nothing),
.RecordDateTime = document.RecordDateTime,
.RestrictedInstrument = (includeRestrictedDocuments AndAlso document.DocumentInstruments.Any() AndAlso document.DocumentInstruments.FirstOrDefault().Instrument.Restricted)
}
I think it is complaining about how the .DocumentClassIds and .InstrumentGroupId's are being loaded into the POCO object (FeeAndReceptionReportIntermediateItem). I would really like to load these up in the initial query, before a .ToList() has been called and I would really like to not even do the join if the user did not pass in the restrictions that require me to create the join, that's why I am using the navigation properties and an if statement when loading these collecctions, because I am assuming if "restrictByDocumentClasses" is false, the navigation property won't be accessed and the join won't be included.
This works as a general pattern. The first line gets an IQueryable<> from the DbSet<>. The select does this for us, so that we can continue reusing query to hold our query as we build it up. Then just keep adding on If...Then...query=query.Where(...)...Endif to continue whittling down the resultset.
var query=db.MyTable.Select(x=>x);
if (RestrictByClasses.Any())
query=query.Where(r =>
r.DocumentClasses.Select(x=>x.ClassId)
.Intersect(RestrictByClasses)
.Any());
if (RestrictBySomethingElse)
query=query.Where(x=>SomethingElse)
I think this is the equivalent in VB.NET:
Dim query = db.MyTable.[Select](Function(x) x)
If RestrictByClasses.Any() Then
query = query.Where(Function(r) r.DocumentClasses.Select(Function(group) group.ClassId).Intersect(RestrictByClasses).Any())
End If
'Repeat as necessary
If RestrictBySomethingElse Then
query = query.Where(Function(x) SomethingElse)
End If
'End repeat
' Rest here is pseudo code
' Sort
SELECT/SWITCH sortonfield
CASE 'name': query=query.OrderBy(Function(x) x.name)
CASE 'dob': query=query.OrderBy(Function(x) x.dob)
DEFAULT: query=query.OrderBy(Function(x) x.id)
END CASE
'Paginate
query=query.Skip((pagenumber-1)*pagesize).Take(pagesize)
'Project
Dim finalresult=query.Select(Function(x) new something {
name=x.Name,
id=x.id,
things=x.things
});
Once all your filters have been put in place (and optionally a sort, and pagination), then project your resultset into whatever you need.
How would I modify the following code:
var result = from p in Cache.Model.Products
from f in p.Flavours
where f.FlavourID == "012541-5-5-5-651"
select p;
So that f.FlavourID is supplied a range of ID's as a supposed to just one value as shown in the above example?
Given the following ERD Model:
Products* => ProdCombinations <= *Flavours
ProdCombinations is a junction/link table and simply has one composite key in there.
Of the top of my head
string [] ids = new[]{"012541-5-5-5-651", "012541-5-5-5-652", "012541-5-5-5-653"};
var result = from p in Cache.Model.Products
from f in p.Flavours
where ids.Contains(f.FlavourID)
select p;
There are some limitations, but an array of ids has worked for me before. I've only actually tried with SQL Server backend, and my IDs were integers.
As I understand it, Linq needs to translate your query into SQL, and it's only possible sometimes. For example it's not possible with IEnumerable<SomeClass>, which produces a runtime error, but possible with a collection of simple types.
In the table ReservationWorkerPeriods there are records of all workers that are planned to work on a given period on any possible machine.
The additional table WorkerOnMachineOnConstructionSite contains columns workerId, MachineId and ConstructionSiteId.
From the table ReservationWorkerPeriods I would like to retrieve just workers who work on selected machine.
In order to retrieve just relevant records from WorkerOnMachineOnConstructionSite table I have written the following code:
var relevantWorkerOnMachineOnConstructionSite = (from cswm in currentConstructionSiteSchedule.ContrustionSiteWorkerOnMachine
where cswm.MachineId == machineId
select cswm).ToList();
workerOnMachineOnConstructionSite = relevantWorkerOnMachineOnConstructionSite as List<ContrustionSiteWorkerOnMachine>;
These records are also used in the application so I don't want to bypass the above code even if is possible to directly retrieve just workerPeriods for workers who work on selected machine. Anyway I haven't figured out how it is possible to retrieve the relevant workerPeriods once we know which userIDs are relevant.
I have tried the following code:
var userIDs = from w in workerOnMachineOnConstructionSite select new {w.WorkerId};
List<ReservationWorkerPeriods> workerPeriods = currentConstructionSiteSchedule.ReservationWorkerPeriods.ToList();
allocatedWorkers = workerPeriods.Where(wp => userIDs.Contains(wp.WorkerId));
but it seems to be incorrect and don't know how to fix it. Does anyone know what is the problem and how it is possible to retrieve just records which contain userIDs from the list?
Currently, you are constructing an anonymous object on the fly, with one property. You'll want to grab the id directly with (note the missing curly braces):
var userIDs = from w in workerOnMachineOnConstructionSite select w.WorkerId;
Also, in such cases, don't call ToList on it - the variable userIDs just contains the query, not the result. If you use that variable in a further query, the provider can translate it to a single sql query.
I want to create an Entity Object from a LinQ statement, but I don't want to load all its columns.
My ORDERS object has a lot of columns, but I just want to retrieve the REFERENCE and OPERATION columns so the SQL statement and result will be smaller.
This LinQ statement works properly and loads all my object attributes:
var orders = (from order in context.ORDERS
select order);
However the following statement fails to load only two properties of my object
var orders = (from order in context.ORDERS
select new ORDERS
{
REFERENCE = order.REFERENCE,
OPERATION = order.OPERATION
});
The error thrown is:
The entity or complex type
'ModelContextName.ORDERS' cannot be
constructed in a LINQ to Entities
query.
What is the problem? Isn't it possible to partially load an object this way?
Thank you in advance for your answers.
ANSWER
Ok I should thank you both Yakimych and Dean because I use both of your answers, and now I have:
var orders = (from order in context.ORDERS
select new
{
REFERENCE = order.REFERENCE,
OPERATION = order.OPERATION,
})
.AsEnumerable()
.Select(o =>
(ORDERS)new ORDERS
{
REFERENCE = o.REFERENCE,
OPERATION = o.OPERATION
}
).ToList().AsQueryable();
And I get exactly what I want, the SQL Statement is not perfect but it returns only the 2 columns I need (and another column which contains for every row "1" but I don't know why for the moment) –
I also tried to construct sub objects with this method and it works well.
No, you can't project onto a mapped object. You can use an anonymous type instead:
var orders = (from order in context.ORDERS
select new
{
REFERENCE = order.REFERENCE,
OPERATION = order.OPERATION
});
The problem with the above solution is that from the moment you call AsEnumerable(), the query will get executed on the database. In most of the cases, it will be fine. But if you work with some large database, fetching the whole table(or view) is probably not what you want. So, if we remove the AsEnumerable, we are back to square 1 with the following error:
The entity or complex type 'ModelContextName.ORDERS' cannot be constructed in a LINQ to Entities query.
I have been struggling with this problem for a whole day and here is what I found. I created an empty class inheriting from my entity class and performed the projection using this class.
public sealed class ProjectedORDERS : ORDERS {}
The projected query (using covariance feature):
IQueryable<ORDERS> orders = (from order in context.ORDERS
select new ProjectedORDERS
{
REFERENCE = order.REFERENCE,
OPERATION = order.OPERATION,
});
Voilà! You now have a projected query that will map to an entity and that will get executed only when you want to.
I think the issue is creating new entities within the query itself, so how about trying this:
context.ORDERS.ToList().Select(o => new ORDERS
{
REFERENCE = o.REFERENCE,
OPERATION = o.OPERATION
});