I want to match in-memory entities to data from DB-tables and return a new in-memory DTO with a subset of that matched information. Now, matching involves two columns, thus I am building a new key on the fly. This works, as long as I execute the queries before building the keys, effectively using Linq-To-Objects for the matching.
When not executing the query right away, I receive a runtime exception as described by this MSDN article.
Here is my code and data model, simplified. I have
Rooms (as IEnumerable<Room>, already in memory)
Areas (as IEnumerable<Room>, already in memory)
Alarms (from the DB, as IQueryable from the context)
Alarms are tied to Areas and LocationIds. Rooms can have multiple Areas, and have one LocationId.
I want to build a set of Alarms occurred in a set of Rooms. This involves matching the Alarm's Area and LocationsId to each Room's LocationId and the Areas.
from area in allAreas
let alarmKey = area.AreaName + area.Room.LocationId //AreaName is String, LocationId is integer
//....
However, this line involves a not supported cast form int to String. How to create the key?
If you don't mind a number of leading spaces in LocationId you can do
let alarmKey = area.AreaName +
SqlFunctions.StringConvert((double)area.Room.LocationId)
SqlFunctions is in System.Data.Objects.SqlClient.
Related
I have built an MVC 5 application, using EF 6 to query the database. One page show a cross table of two dimensions: substances against properties of these substances. It is rendered as an html table.Many cells do not have a value. This is what it looks like:
sub 1 sub 2 sub 3
prop A 1.0
prop B 1.5 X
prop C 0.6 Y
The cell values are actually more complex, including tool tips, footnotes, etc.
I implemented the generation of the html table, by the following steps:
create a list of unique properties;
create a list of unique substances;
loop through the properties;
render a row for each;
loop through the substances;
See if there is a value for the combination of property and substances;
render the cell's value or an empty one.
Using the ANTS performance profiler, I found out that step 6 has a huge performance issue with increasing numbers of substances and properties, the hit count exploding to hundreds of millions, with a few hundred substances and a few tens of properties (the largest selection the user can make). The execution time is many minutes. It seems to scale N(substances)^2 * N(properties)^2.
The code looks like:
Value currentValue =
values.Where(val => val.substance.Id == currentSubstanceId
&& val.property.Id == currentPropertyId).SingleOrDefault();
where values is a List and Value is an entity, which I read from to render the cells. values had been pre-loaded from the database and no queries are shown by the SQL Server Profiler.
Since not all cells have a value, I thought it best to loop through the row and columns and see if there is a value. I cannot just loop through the list of values.
What could I try to improve this? I thought about:
Create some sort of C# object, using the substance.Id and property.Id as a compound key and fill it from the List object. Which would be fastest?
Create some Linq query which returns an object which already contains the empty cells, like (substance cross join properties) left join values. I could do this in SQL easily, but could this be done with Linq? Could the object which stores the result have the Value as a member field, so I can still use it to render the cells?
Stop pre-loading and just run a database query for the value of each combination, possibly benefiting from database indexes.
I am considering restricting the number of substances and properties the user may select, but I would rather not do that.
Addtional info
As requested by C.Zonnenberg, some more info about the query.
The query to fill the list of values is basically as follows:
I create an IQueryable to which I add filters for requested substances and properties. I then include the substances, property and value details, found in related entities. I then execute query.ToList(). The actual SQL query, as seen by the SQL Profiler looks complex, involving SubstanceId IN () and PropertyId IN (), but it takes far less then a second to execute.
It returns a list of proxies, like: {System.Data.Entity.DynamicProxies.SubstancePropertyValue_078F758A4FF9831024D2690C4B546F07240FAC82A1E9D95D3826A834DCD91D1E}
I think your best bet is your first option. But to do that efficiently I would also modify the source data (values) and turn it into a dictionary, so you have a structure that's optimized for indexed lookup:
var dict = values.ToDictionary(e =>
Tuple.Create(e.substance.id, e.propertyid),
e => e.Value);
Then for each cell:
Value currentValue ;
dict.TryGetValue(Tuple.Create(currentSubstanceId, currentPropertyId),
out currentValue );
Further, you may benefit from parallelization by fetching the cell values in a Parallel.ForEach looping through all substances, for instance.
Suppose I have a large (300-500k) collection of text documents stored in the relational database. Each document can belong to one or more (up to six) categories. I need users to be able to randomly select documents in a specific category so that a single entity is never repeated, much like how StumbleUpon works.
I don't really see a way I could implement this using slow NOT IN queries with large amount of users and documents, so I figured I might need to implement some custom data structure for this purpose. Perhaps there is already a paper describing some algorithm that might be adapted to my needs?
Currently I'm considering the following approach:
Read all the entries from the database
Create a linked list based index for each category from the IDs of documents belonging to the this category. Shuffle it
Create a Bloom Filter containing all of the entries viewed by a particular user
Traverse the index using the iterator, randomly select items using Bloom Filter to pick not viewed items.
If you track via a table what entries that the user has seen... try this. And I'm going to use mysql because that's the quickest example I can think of but the gist should be clear.
On a link being 'used'...
insert into viewed (userid, url_id) values ("jj", 123)
On looking for a link...
select p.url_id
from pages p left join viewed v on v.url_id = p.url_id
where v.url_id is null
order by rand()
limit 1
This causes the database to go ahead and do a 1 for 1 join, and your limiting your query to return only one entry that the user has not seen yet.
Just a suggestion.
Edit: It is possible to make this one operation but there's no guarantee that the url will be passed successfully to the user.
It depend on how users get it's random entries.
Option 1:
A user is paging some entities and stop after couple of them. for example the user see the current random entity and then moving to the next one, read it and continue it couple of times and that's it.
in the next time this user (or another) get an entity from this category the entities that already viewed is clear and you can return an already viewed entity.
in that option I would recommend save a (hash) set of already viewed entities id and every time user ask for a random entity- randomally choose it from the DB and check if not already in the set.
because the set is so small and your data is so big, the chance that you get an already viewed id is so small, that it will take O(1) most of the time.
Option 2:
A user is paging in the entities and the viewed entities are saving between all users and every time user visit your page.
in that case you probably use all the entities in each category and saving all the viewed entites + check whether a entity is viewed will take some time.
In that option I would get all the ids for this topic- shuffle them and store it in a linked list. when you want to get a random not viewed entity- just get the head of the list and delete it (O(1)).
I assume that for any given <user, category> pair, the number of documents viewed is pretty small relative to the total number of documents available in that category.
So can you just store indexed triples <user, category, document> indicating which documents have been viewed, and then just take an optimistic approach with respect to randomly selected documents? In the vast majority of cases, the randomly selected document will be unread by the user. And you can check quickly because the triples are indexed.
I would opt for a pseudorandom approach:
1.) Determine number of elements in category to be viewed (SELECT COUNT(*) WHERE ...)
2.) Pick a random number in range 1 ... count.
3.) Select a single document (SELECT * FROM ... WHERE [same as when counting] ORDER BY [generate stable order]. Depending on the SQL dialect in use, there are different clauses that can be used to retrieve only the part of the result set you want (MySQL LIMIT clause, SQLServer TOP clause etc.)
If the number of documents is large the chance serving the same user the same document twice is neglibly small. Using the scheme described above you don't have to store any state information at all.
You may want to consider a nosql solution like Apache Cassandra. These seem to be ideally suited to your needs. There are many ways to design the algorithm you need in an environment where you can easily add new columns to a table (column family) on the fly, with excellent support for a very sparsely populated table.
edit: one of many possible solutions below:
create a CF(column family ie table) for each category (creating these on-the-fly is quite easy).
Add a row to each category CF for each document belonging to the category.
Whenever a user hits a document, you add a column with named and set it to true to the row. Obviously this table will be huge with millions of columns and probably quite sparsely populated, but no problem, reading this is still constant time.
Now finding a new document for a user in a category is simply a matter of selecting any result from select * where == null.
You should get constant time writes and reads, amazing scalability, etc if you can accept Cassandra's "eventually consistent" model (ie, it is not mission critical that a user never get a duplicate document)
I've solved similar in the past by indexing the relational database into a document oriented form using Apache Lucene. This was before the recent rise of NoSQL servers and is basically the same thing, but it's still a valid alternative approach.
You would create a Lucene Document for each of your texts with a textId (relational database id) field and multi valued categoryId and userId fields. Populate the categoryId field appropriately. When a user reads a text, add their id to the userId field. A simple query will return the set of documents with a given categoryId and without a given userId - pick one randomly and display it.
Store a users past X selections in a cookie or something.
Return the last selections to the server with the users new criteria
Randomly choose one of the texts satisfying the criteria until it is not a member of the last X selections of the user.
Return this choice of text and update the list of last X selections.
I would experiment to find the best value of X but I have in mind something like an X of say 16?
I'm trying to do something a little clever with my app. I have a table, Adverts - which contains info on cars: model, mileage etc. The table is related to a few other tables via foreign keys e.g. model name is retrieved through a foreign key linking to a "VehicleModels" table etc.
Within the app's "Entities" dir (classes which map to tables in the database) I have one for the Adverts table, Advert.cs. This has a couple of properties which EF has been told to ignore (in fluent api) as they don't map to actual fields in the Adverts table.
The idea behind these fields is to store the calculated distance from a postcode (zip code) the user enters in a search form which filters through the Adverts table if they only want to see cars available within a certain radius. e.g.:
IQueryable<Advert> FilteredAdverts = repository.Adverts
.Where(am => mfr == "" || am.Manufacturer == mfr) &&
(am => model == etc etc...)
Later on, to calculate the distance the code resembles:
if (userPostcode != null) {
foreach (var ap in FilteredAdverts.ToList()) {
distmiles = //calculate distance in miles
distkm = //calculate distance in km
ap.DistanceMiles = Convert.ToInt32(distmiles);
ap.DistanceKm = Convert.ToInt32(distkm);
}
}
The problem I'm having is that in order to assign values to these two fields, I'm having to use .ToList() which is pulling all rows from the table. Works ok if there are only a few rows, but when there are ~1,000 it takes approx. 2.2 seconds, when I increased it to about 12,000 rows it took 32 seconds for the page to load when no filters were applied i.e. all active adverts returned.
The reason I'm pulling all adverts before calling .Skip and .Take to display them is that the filters available in the search form are based on possible options of all current adverts that are active i.e. have time remaining, rather than just selecting a list of manufacturers from the manufacturers table (where a user could choose a manufacturer for which there are no search results). e.g.
VehicleManufacturers = (from vm in FilteredAdverts.Select(x => x.VehicleManufacturer).Distinct().OrderBy(x => x)
select new SearchOptionsModel
{
Value = vm,
Text = vm,
Count = FilteredAdvertsVM.Where(x => x.VehicleManufacturer == vm).Count(),
})
.... filters for model, mileage etc
To get an idea of what I'm trying to achieve - take a look at the search form on the Autotrader website.
Once all the filters are applied, just before the model is passed to the view, .Skip and .Take are applied, but of course by this time all rows have been pulled.
My question is, how do I go about redoing this? Is there a better method to make use of these non-mapped properties in my Advert entity class? I'm working on my home PC - C2D # 3.4GHz, 2GB ram - would the slow queries run ok on a propert web host ?
You cannot use server-side paging on a client side function. That's the short answer. Assuming I understand your need correctly (to filter a list based on proximity to a given zip code), a solution I've used in the past is storing each 'Advert' record with a lat/long for that record's zip code. This data is persisted.
Then, when it comes time to query, calculate a bounding box (lat1, lng1, lat2, lng2) based on X distance from the center (user provided zip code) and filter the query results based on records whose lat/lng fits within this box. You can then apply client side calculations to further and more accurately filter the list, but using this method, you can establish a base filter to minimize the number of records pulled.
Edit: You can order the results of the query based on the absolute distance from the center point in terms of abs(latU-latR) and abs(lngU-lngR) where latU/lngU is the lat/lng of the user provided zip code and latR/lngR is the lat/lng of the record in the db.
I'm fairly new to the more complex parts of Core Data.
My application has a core data store with 15K rows. There is a single entity.
I need to display a subset of those rows in a table view filtered on a calculated search criteria, and for each row displayed add a value that I calculate in real time but don't store in the entity.
The calculation needs to use a couple of values supplied by the user.
A hypothetical example:
Entity: contains fields "id", "first", and "second"
User inputs: 10 and 20
Search / Filter Criteria: only display records where the entity field "id" is a prime number between the two supplied numbers. (I need to build some sort of complex predicate method here I assume?)
Display: all fields of all records that meet the criteria, along with a derived field (not in the the core data entity) that is the sum of the "id" field and a random number, so each row in the tableview would contain 4 fields:
"id", "first", "second", -calculated value-
From my reading / Googling it seems that a transient property might be the way to go, but I can't work out how to do this given that the search criteria and the resultant property need to calculate based on user input.
Could anyone give me any pointers that will help me implement this code? I'm pretty lost right now, and the examples I can find in books etc. don't match my particular needs well enough for me to adapt them as far as I can tell.
Thanks
Darren.
The first thing you need to do is to stop thinking in terms of fields, rows and columns as none of those structures are actually part of Core Data. In this case, it is important because Core Data supports arbitrarily complex fetches but the sqlite store does not. So, if you use a sqlite store your fetches are restricted those supported by SQLite.
In this case, predicates aimed at SQLite can't perform complex operations such as calculating whether an attribute value is prime.
The best solution for your first case would be to add a boolean attribute of isPrime and then modify the setter for your id attribute to calculate whether the set id value is prime or not and then set the isPrime accordingly. That will be store in the SQLite store and can be fetched against e.g. isPrime==YES &&((first<=%#) && (second>=%#))
The second case would simply use a transient property for which you would supply a custom getter to calculate its value when the managed object was in memory.
One often overlooked option is to not use an sqlite store but to use an XML store instead. If the amount of data is relatively small e.g. a few thousand text attributes with a total memory footprint of a few dozen meg, then an XML store will be super fast and can handle more complex operations.
SQLite is sort of the stunted stepchild in Core Data. It's is useful for large data sets and low memory but with memory becoming ever more plentiful, its loosing its edge. I find myself using it less these days. You should consider whether you need sqlite in this particular case.
1)
public class Query<T> : IQueryable<T> ...
{
...
public IEnumerator<T> GetEnumerator()
{
return((IEnumerable<T>)this.provider.Execute(this.expression)).GetEnumerator();
}
}
Query<string> someQuery = new Query<string>();
someQuery.Expression holds the expression tree associated with this particular instance of IQueryable and thus describes how to retrieve the initial sequence of items. In the following example initialSet variable actually contains a sequence of strings:
string[] initialSet = { };
var results1 = from x in initialSet
where ...
select ...;
a) Can someQuery also contain a sequence of strings, or can it only contain instructions ( in the form of expression tree ) on how to get this initial sequence of strings from some DB?
b) I assume that even if someQuery actually contains an initial sequence of strings, it is of no use to Where and Select operators, since they won't ever operate on this sequence of strings, but instead their job is only to build queries or request queries to be executed ( by calling IQueryProvider.Execute)? And for this reason someQuery must always contain an expression tree describing how too get the initial sequence of strings, even if someQuery already contains this initial sequence?
Thank you
EDIT:
c) The way I understood your post is that query provider may contain information about describing the table or at least describing particular DB rows which initial query needs to retrieve. But I didn't interpret your answer as saying that query provider may also contain actual elements required by this initial query ( someQuery in our example )?
d) Regardless, I assume even if query provider maintains actual elements, it can only maintain them for initial query? Thus if we apply Linq-to-entity or Linq-to-Sql operators on that initial query, I assume provider will have to query the database. As such, if my assumption are correct then answer to b) would be even if query does contain actual elements, when we call Where on someQuery ( someQuery.Where ),query provider will have to retrieve results from a DB, even if this query provider already contains all the elements of someQuery?
e) I only started learning Linq-to-entities, so my question may be too general, but how does EF handle all of this? In other words, when does ObjectSet<T> returned by some EF API ( such as ObjectContext ), contain actual elements and when does it ( if ever ) contain only logic for retrieving elements from some data source (such as DB)?
f) Also, even if ObjectSet<T> ( returned by say ObjectSet ) does contain actual elements, I assume if we apply Where operator on it ( ObjectSet<T>.Where ), query provider will always have to retrieve results from the DB?
a) You wouldn't normally create a Query<T> yourself - a query provider would. It can choose to include whatever information it wants. It's most likely to just contain the information about what table it's associated with though.
b) That's entirely up to the query provider. As you saw in the other question, the query provider may end up recognizing when it's reached a Query<T> - so it could know to ask the Query<T> for its strings, if that was appropriate.
c) The query provider wouldn't usually contain data itself - but it could do. It's up to the provider.
d) The query provider may notice that it's within a transaction, and that it's already executed a similar context in the query - it may be able to answer the query from within its cache. It's up to the query provider.
e, f) No idea, I've never used Entity Framework in anger.
The answer to almost all of your questions around this is topic is "it's up to the query provider". For details about a particular query provider, you should read the documentation for that provider. That should explain when it will make queries etc. It's unclear what you're really trying to get out of these questions - but if you're after a full implementation to study, there are plenty of open source LINQ providers around. You might want to look at NHibernate for example.