Using MongoDB to store time series data of arbitrary intervals - performance

Question

I want to store time-series-like data. There are no set intervals for the data like normal time series data. Data points could be as often as every few seconds to as seldom as every few years, all in the same time series. I basically need to store the Date data type and a value, over and over.
I would like the ability to very quickly retrieve the most recent item in the series. I would also like the ability to quickly retrieve all the values within a range between two dates. Writing efficiency is nice but not as important.
My initial thought was to use documents with keys set to dates. Something like this:
{
"entry_last": 52,
"entry_history": {
datetime(2013, 1, 15): 94,
datetime(2014, 12, 23): 25,
datetime(2016, 10, 23, 5, 34, 00): 52
}
}
However, from my undertstanding, keys have to be strings.
So then I came up with this prototype:
{
"entry_last": 52,
"entry_history": [
[datetime(2013, 1, 15), 94],
[datetime(2014, 12, 23), 25],
[datetime(2016, 10, 23, 5, 34, 00), 52],
]
}
The idea here is to give myself very easy access to the last value with entry_last (the value of which is duplicated in the history), as well as to store each data entry in the most efficient way possible by only storing the date and value in entry_history.
What I'd like to know is whether or not my prototype is an efficient approach to storing my data. Specifically, I'd like to know if this will allow me to efficiently query the most recent value as well as values between two dates. If not, what is a better approach?

Answer 1

You don't have to manually specify the index, you can store only the datetime and use the index of the array.
The main issue I see with your solution is you have to manually maintain entry_last, if the update ever fails, this doesn't work anymore, unless you have few failsafes. If you build another app with a different technology using the same db, you'll have to recode the same logic. And I don't see how to query between two dates easily and efficiently here, unless you reorder the array every time you insert an element.
If I had to design this kind of data storing, I would create another collection to store the history (linked to your entries by _id) and index the date to fast query. But it might depend on the quantity of your data.
/* entry */
{
_id: 1234,
"entryName": 'name'
}
/* history */
{
_id: 9876,
"_linkedEntryId": 1234,
"date": new Date(2013, 1, 15)
}
{
_id: 9877,
"_linkedEntryId": 1234,
"date": new Date(2014, 12, 23)
}
{
_id: 9878,
"_linkedEntryId": 1234,
"date": new Date(2016, 10, 23, 5, 34, 00)
}
To give an idea of the performance, I have a mongodb running on my ultrabook (far from a dedicated server's performance) and I can get the most recent document linked to a specific identifier in 5-10ms. Same speed to get all documents between two dates. I'm querying a modest collection of one million documents. It's not random data, the average object's size is 2050B.