Populating child collection property with LINQ sub-query - linq

Question

I'm fairly new to LINQ in general but I've managed to figure things out so far to get the result I want/need. The LINQ2SQL query below produces the desired results in which there are Location objects, each with their own collection of Units. The initial variable declarations are done to setup the values that I need to query from and should not be relevant to my issue but are needed for context.
The problem I'm having is that there are 400+ locations, and the current way I have the query structured, it is hitting the database for each one instead of one big query. This is causing performance to suffer greatly, taking 30+ seconds for the full result to come back which is unacceptable. I have a stored procedure SQL query that returns the results in only a second or two.
Is there a way to restructure this query so that I'm not hitting the database for each location? I'm sure it's caused by the extra "ToList()" on the Units subquery but I don't know how to shape the result without the ToList(). Any performance tweaking would be a HUGE help. Thanks for any help!
Dim curNames1 = (From ons In dc.organization_names _
Where ons.eff_date <= ssDate _
Order By ons.eff_date Descending _
Group ons By ons.organization_id Into gNames = Group _
Select New With { _
Key .Key = organization_id, _
.Result = gNames.Take(1)}) _
.SelectMany(Function(a) (a.Result))
Dim curLocs1 = (From oLocs In dc.organization_locations _
Where oLocs.eff_date <= ssDate _
Order By oLocs.eff_date Descending _
Group oLocs By oLocs.organization_id Into gLocs = Group _
Select New With { _
Key .Key = organization_id, _
Result = gLocs.Take(1)}) _
.SelectMany(Function(a) (a.Result))
Dim curStatuses1 = (From oEDate In dc.organization_conversion_dates _
Where oEDate.edate <= ssDate _
Order By oEDate.edate Descending _
Group oEDate By oEDate.organization_id Into gEDates = Group _
Select New With { _
Key .Key = organization_id, _
.Result = gEDates.Take(1)}) _
.SelectMany(Function(a) (a.Result))
Dim curCommand1 = (From oCommand In dc.organization_service_assigneds _
Where oCommand.eff_date <= ssDate _
Order By oCommand.eff_date Descending _
Group oCommand By oCommand.organization_id Into gCmds = Group _
Select New With { _
Key .Key = organization_id, _
.Result = gCmds.Take(1)}) _
.SelectMany(Function(a) (a.Result))
Dim curComponent1 = (From oCompo In dc.organization_compos _
Where oCompo.eff_date <= ssDate _
Order By oCompo.eff_date Descending _
Group oCompo By oCompo.organization_id Into gCompos = Group _
Select New With { _
Key .Key = organization_id, _
.Result = gCompos.Take(1)}) _
.SelectMany(Function(a) (a.Result))
ss.Locations = New ObservableCollection(Of Location)(( _
From map In dc.MapBackgrounds Where map.MapID = options.Map _
Let Locs = map.Schemes.SchemeLocations _
From SchemeLoc In Locs _
Where (options.Locations.Count = 0 Or _
options.Locations.Contains(SchemeLoc.Location.ID)) _
Select New Location With { _
.ID = SchemeLoc.Location.ID, .Name = SchemeLoc.Location.Location, _
.Y = SchemeLoc.Y, .X = SchemeLoc.X, _
.Country = New Country With _
{.ID = SchemeLoc.Location.Countries.ID, _
.Name = SchemeLoc.Location.Countries.country}, _
.State = If(SchemeLoc.Location.State = -1, Nothing, _
New USState With {.ID = SchemeLoc.Location.States.ID, _
.Name = SchemeLoc.Location.States.state, _
.Abbreviation = SchemeLoc.Location.States.state}), _
.Units = New ObservableCollection(Of MillitaryUnit)(( _
From curLoc In curLocs1 _
Where curLoc.location = SchemeLoc.Location.ID _
From curName In curNames1 _
Where curName.organization_id = curLoc.organization_id _
And (options.UnitSizes.Count = 0 Or _
options.UnitSizes.Contains(curName.UnitSize)) _
And (options.UnitTypes.Count = 0 Or _
options.UnitTypes.Contains(curName.UnitType)) _
From curEDate In curStatuses1 _
Where curEDate.organization_id = curLoc.organization_id _
And (options.Statuses.Count = 0 Or _
options.Statuses.Contains(curEDate.status)) _
From curCmd In curCommand1 _
Where curCmd.organization_id = curLoc.organization_id _
And (options.Commands.Count = 0 Or _
options.Commands.Contains(curCmd.service_assigned)) _
From curCompo In curComponent1 _
Where curCompo.organization_id = curLoc.organization_id _
And (options.Components.Count = 0 Or _
options.Components.Contains(curCompo.compo)) _
From curTable In curLoc.Organization.organization_tables _
Where curTable.organization_id = curLoc.organization_id _
And (options.Tables.Count = 0 Or _
(options.Tables.Contains(curTable.table_id) Or _
curTable.Tables.Any(Function(a) (options.Tables.Contains(a.parent_id))))) _
Select New MillitaryUnit With { _
.ID = curLoc.organization_id, _
.Name = curName.name, _
.IconPath = curName.icon, _
.ConversionStatus = curEDate.Status1.status, _
.ServiceCommand = curCmd.Service_Assigneds.service_assigned, _
.Component = curCompo.Components.compo}).Distinct().ToList())}).ToList())
UPDATE (3/3/2009 10:58 AM):
I managed to get the data back in one query using the query below but it's a flat table result. How can I shape this so that the organization stuff becomes hierarchical under each location? I believe I want to use something like "Group Join" but I'm not familiar with how that works. The location info is everything up to the "OrgID" column. I need to shape this data into a collection of Locations that each have a "Units" property which is a collection of the organizations that are at that location. Any guidance?
Dim locationsquery = (From map In dc.MapBackgrounds Where map.MapID = 1 Let Locs = map.Schemes.SchemeLocations _
From SchemeLoc In Locs Join curLoc In curLocs1 On SchemeLoc.LocID Equals curLoc.location _
Join curName In curNames1 On curLoc.organization_id Equals curName.organization_id _
Join curStatus In curStatuses1 On curLoc.organization_id Equals curStatus.organization_id _
Join curCommand In curCommand1 On curLoc.organization_id Equals curCommand.organization_id _
Join curComponent In curComponent1 On curLoc.organization_id Equals curComponent.organization_id _
Select New With {.LocationID = SchemeLoc.LocID, .X = SchemeLoc.X, .Y = SchemeLoc.Y, _
.LocationName = SchemeLoc.Location.Location, _
.CountryID = SchemeLoc.Location.Countries.id, .CountryName = SchemeLoc.Location.Countries.country, _
.StateID = SchemeLoc.Location.State, .StateName = SchemeLoc.Location.States.state, .StateAbbrev = SchemeLoc.Location.States.state, _
.OrgID = curLoc.organization_id, _
.OrgName = curName.name, _
.OrgLocID = curLoc.location, _
.IconPath = curName.icon, _
.ConversionStatus = curStatus.status1.status, _
.CurCmd = curCommand.service_assigneds.service_assigned, _
.CurCompo = curComponent.components.compo})

Answer 1

Instead of exposing the member as a List<T>, change the property type to an IEnumerable<T> or an IQueryable<T>. Then, you can remove the calls to ToList and your query should be able to be executed completely on the server in one shot.
Also, this kind of code is where LINQ to SQL starts to break down, IMO. I think that if you have a stored procedure which will give you the desired results, then you should use that, and just have LINQ to SQL handle the mapping of the returned data to your objects.

Answer 2

So I ended up redoing things and came up with the following solution which only queries the database twice and is MUCH faster:
Dim locationsOnly = (From map In dc.MapBackgrounds Where map.MapID = 1 Let Locs = map.Schemes.SchemeLocations _
From SchemeLoc In Locs _
Where (options.Locations.Count = 0 Or options.Locations.Contains(SchemeLoc.LocID)) _
Select New With {.LocationID = SchemeLoc.LocID, .X = SchemeLoc.X, .Y = SchemeLoc.Y, _
.LocationName = SchemeLoc.Location.Location, _
.CountryID = SchemeLoc.Location.Countries.ID, .CountryName = SchemeLoc.Location.Countries.country, _
.StateID = SchemeLoc.Location.State, .StateName = SchemeLoc.Location.States.state, .StateAbbrev = SchemeLoc.Location.States.state}).ToList()
Dim orgsOnly = (From curLoc In curLocs1 _
Join curName In curNames1 On curLoc.organization_id Equals curName.organization_id _
Join curStatus In curStatuses1 On curLoc.organization_id Equals curStatus.organization_id _
Join curCommand In curCommand1 On curLoc.organization_id Equals curCommand.organization_id _
Join curComponent In curComponent1 On curLoc.organization_id Equals curComponent.organization_id _
Join curTable In dc.organization_tables On curLoc.organization_id Equals curTable.organization_id _
Where (options.UnitSizes.Count = 0 Or options.UnitSizes.Contains(curName.UnitSize)) _
And (options.UnitTypes.Count = 0 Or options.UnitTypes.Contains(curName.UnitType)) _
And (options.Statuses.Count = 0 Or options.Statuses.Contains(curStatus.status)) _
And (options.Commands.Count = 0 Or options.Commands.Contains(curCommand.service_assigned)) _
And (options.Components.Count = 0 Or options.Components.Contains(curComponent.compo)) _
And (options.Tables.Count = 0 Or (options.Tables.Contains(curTable.table_id) Or curTable.Tables.Any(Function(a) (options.Tables.Contains(a.parent_id))))) _
Select New With {.OrgLocID = curLoc.location, _
.MilUnit = New MillitaryUnit With { _
.ID = curLoc.organization_id, _
.Name = curName.name, _
.IconPath = curName.icon, _
.ConversionStatus = curStatus.Status1.status, _
.ServiceCommand = curCommand.Service_Assigneds.service_assigned, _
.Component = curComponent.Components.compo}}).Distinct().ToList()
ss.Locations = New System.Collections.ObjectModel.ObservableCollection(Of Location)((From loc In locationsOnly _
Group Join org In orgsOnly On loc.LocationID Equals org.OrgLocID Into Orgs = Group _
Select New Location With { _
.ID = loc.LocationID, _
.Name = loc.LocationName, _
.X = loc.X, _
.Y = loc.Y, _
.Country = New Country With {.ID = loc.CountryID, .Name = loc.CountryName}, _
.State = New USState With {.ID = loc.StateID, .Name = loc.StateName, .Abbreviation = loc.StateAbbrev}, _
.Units = New System.Collections.ObjectModel.ObservableCollection(Of MillitaryUnit)((From curOrg In Orgs _
Select curOrg.MilUnit).ToList())}).ToList())
Thanks for your help casperOne!