most efficient method for combining query results - linq

Question

This question is somewhat exploratory as I decided to ask for advice/suggestions prior to attempting implementation in the hopes of a more timely path to the result. I have a database that stores intraday data for commodities and equities in 1 minute formats. The rationale behind this was that if I had 1 minute bars, I could create any time series bar that I wanted(i.e., 5 min , 15 min, 60 min etc,etc)
I thought about going the route where, for example, on a 15 minute chart I would simply iterate through the whole subset of results for a given symbol, look for rows with timestamps of :01,:16,:31:46 to begin capturing the open, close, maxhigh and minlow and cumulative volume and create new datapoint from that amalgamation. The problem with that is that, for commodities especially that trade 24 hours, there is not always a bar for every minute so there COULD be a situation where there isn't a :16 bar, or a :31 bar for a given symbol. This could foul the whole sequence of getting the bars data correctly. This also eliminated the possibility of just grabbing 15 bars at a time and capturing the open, close, maxhigh and minlow and cumulative volume. (Naturally, the table contains symbol, datetime, open,high,low,close,volume)
In order for either of the above to work I would have to "fix" the database every night by checking for missing bars and copying the prior bar to create a bar for that missing row. This is not the preferred method but Ill make it happen if necessary.
I am looking for any guidance on a proper path, whether from past experience or from reading about this problem here.
UPDATE:
Here is the code I've come up with that works BUT it is a little slow, it takes approximately 30 seconds to create 2 months of 30 minute bars from the 1 minute data in the database table.
Partial Class testintracharts
Inherits Page
<WebMethod>
Public Shared Function GetBars(ByVal symbol As String, ByVal seriesInterval As Integer) As List(Of ArrayList)
Dim barsList As New List(Of ArrayList)
'replace with TD model
Using ctx As New BATLEntities()
ctx.Configuration.AutoDetectChangesEnabled = False
'get all days for a given symbol, ordered from oldest to newest
Dim dateList As List(Of DateTime) = GetDistinctDates(ctx, symbol, seriesInterval).ToList()
'cache data series toa list for the given symbol
Dim seriesList As List(Of tsintrachart) = GetSymbolSeries(ctx, symbol)
If Not dateList Is Nothing And dateList.Any() Then
For Each seriesDate As DateTime In dateList
Dim curTime As TimeSpan = New TimeSpan(0, 1, 0)
Dim maxTime As TimeSpan = New TimeSpan(24, 0, 0)
'loop through the data series for the given day
'series start at 00:01:00
'series ends at 00:00:00 (next day)
While curTime < maxTime
Dim seriesMax As TimeSpan = curTime.Add(New TimeSpan(0, seriesInterval - 1, 0))
'get the data chunk based on series interval
'special condition when seriesmax reaches 24:00:00, in TimeSpan this becomes 1.00:00:00 (SQL doesn't like this)
'query needs to incorporate 00:00:00 of next day as the last entry for this time series
Dim data As List(Of tsintrachart) = Nothing
If TimeSpan.Compare(seriesMax, maxTime) = 0 Then
Dim nextSeriesDate As DateTime = seriesDate.AddDays(1)
data = GetDataSeries(seriesList, seriesDate, curTime, nextSeriesDate, New TimeSpan(0, 0, 0))
Else
data = GetDataSeries(seriesList, seriesDate, curTime, seriesDate, seriesMax)
End If
If Not data Is Nothing And data.Any() Then
Dim lastbarnum As Integer = data.Count - 1
Dim intradayDatum As New ArrayList()
With intradayDatum
.Add(DateTimeToUnixTimestamp(seriesDate.Add(seriesMax))) 'date
.Add(data(0).Open)
'.Add((From d In data Where TimeSpan.Compare(d.Time, curTime) = 0 Select d.Open).FirstOrDefault()) 'open
.Add((From d In data Select d.High).Max()) 'high
.Add((From d In data Select d.Low).Min()) 'low
.Add(data(lastbarnum).Close)
'.Add((From d In data Where TimeSpan.Compare(d.Time, seriesMax) = 0 Select d.Close).FirstOrDefault()) 'close or last sale
.Add((From d In data Select d.Volume).Sum()) 'volume
End With
barsList.Add(intradayDatum)
End If
'update current series start time, move to next series chunk
curTime = curTime.Add(New TimeSpan(0, seriesInterval, 0))
End While
Next
End If
End Using
Return barsList
End Function
Private Shared Function GetDataSeries(ByRef list As List(Of tsintrachart), ByVal startDate As DateTime, ByVal startSpan As TimeSpan, ByVal endDate As DateTime, ByVal endSpan As TimeSpan) As List(Of tsintrachart)
'LINQ, where symbol/time>=startRange/time<=endRange
Dim series = From data In list
Where (TimeSpan.Compare(data.Time, startSpan) >= 0 AndAlso data.Date = startDate) _
And (TimeSpan.Compare(data.Time, endSpan) <= 0 AndAlso data.Date = endDate)
Select data
Return series.ToList()
End Function
Private Shared Function GetSymbolSeries(ByRef ctx As BATLEntities, ByVal symbol As String) As List(Of tsintrachart)
Dim series = From data In ctx.tsintracharts
Where (data.Symbol = symbol)
Select data
Return series.ToList()
End Function
Private Shared Function GetDistinctDates(ByRef ctx As BATLEntities, ByVal symbol As String, ByVal interval As Integer) As IQueryable(Of DateTime)
Dim numDates As Integer = TakeValue(interval)
Dim dates = (From data In ctx.tsintracharts
Where data.Symbol = symbol
Select data.Date
Order By [Date]).Distinct().Take(numDates)
Return dates
End Function
Private Shared Function TakeValue(ByVal interval As Integer) As Integer
Select Case interval
Case 0 To 15
Return 1440
Case 16 To 30
Return 720
Case 31 To 60
Return 528
Case Else
Return 400
End Select
End Function
End Class

Answer 1

You could create a table that holds all possible 1min points in time. Then you can OUTER JOIN to that table to make the server fill in any gaps. The table would be like this:
CREATE TABLE TimePoints (
DateTime DATETIME2(0) PRIMARY KEY
)
And you have to fill it will a few decades of data. To query, you would join to it:
SELECT *
FROM TimePoints
LEFT JOIN myOtherTable ON ...
WHERE TimePoints.DateTime >= (nowMinus15min) AND TimePoints.DateTime <= now