How to calculate real-time stats? - performance

Question

I have a site with millions of users (well, actually it doesn't have any yet, but let's imagine), and I want to calculate some stats like "log-ins in the past hour".
The problem is similar to the one described here: http://highscalability.com/blog/2008/4/19/how-to-build-a-real-time-analytics-system.html
The simplest approach would be to do a select like this:
select count(distinct user_id)
from logs
where date>='20120601 1200' and date <='20120601 1300'
(of course other conditions could apply for the stats, like log-ins per country)
Of course this would be really slow, mainly if it has millions (or even thousands) of rows, and I want to query this every time a page is displayed.
How would you summarize the data? What should go to the (mem)cache?
EDIT: I'm looking for a way to de-normalize the data, or to keep the cache up-to-date. For example I could increment an in-memory variable every time someone logs in, but that would help to know the total amount of logins, not the "logins in the last hour". Hope it's more clear now.

Answer 1

IMO the more correct approach here would be to implement a continuous calculation that holds the relevant counters in memory. Every time a user is added to your system you can fire up an event which can be processed in multiple ways and update last hour, last day or even total users counters. There are some great frameworks out there to do this sort of processing. Twitter Storm is one of them, another one is GigaSpaces XAP (disclaimer - I work for GigaSpaces) and specifically this tutorial, and also Apache S4 and GridGain.

Answer 2

If you don't have a db then never mind. I don't have millions of users but I have table with a years worth of logon that has a million rows and simple stats like that in sub second. A million rows is not that much for a database. You cannot make date a PK as you can have duplicates. For minimal fragmentation and speed of insert make date a clustered non unique index asc and that is how the data comes in. Not sure if you have a DB but in MSSQL you can. Index user_id is something to test. What that would do is slow down insert as that is an index that will fragment. If you a looking for a fairly tight time span a table scan might be OK.
Why distinct user_id rather then a login is a login.
Have a property that only only runs the query every x seconds. Even if every second and reports the cached answer. If or 200 pages hit that property in one second for sure you don't want 200 queries. And if the stat is one second stale for information over the last hour that is still a valid stat.

Answer 3

I've ended up using Esper/NEsper. Also Uri's suggestions where useful.
Esper allows me to compute real-time stats of data as it's being obtained.

Answer 4

If you're just running off of logs, you probably want to look at something like Splunk.
Generally, if you want this in-memory and fast (real time), you would create a distributed cache of login data with an eviction after e.g. 24 hours, and then you could query that cache for e.g. logins within the past hour.
Assuming a login record looks something like:
public class Login implements Serializable {
public Login(String userId, long loginTime) {..}
public String getUserId() {..}
public long getLoginTime() {..}
public long getLastSeenTime() {..}
public void setLastSeenTime(long logoutTime) {..}
public long getLogoutTime() {..}
public void setLogoutTime(long logoutTime) {..}
String userId;
long loginTime;
long lastSeenTime;
long logoutTime;
}
To support the eviction after 24 hours, simply configure an expiry (TTL) on the cache
<expiry-delay>24h</expiry-delay>
To query for all users currently logged in:
long oneHourAgo = System.currentTimeMillis() - 60*60*1000;
Filter query = QueryHelper.createFilter("loginTime > " + oneHourAgo
+ " and logoutTime = 0");
Set idsLoggedIn = cache.keySet(query);
To query for the number of logins and/or active users in the past hour:
long oneHourAgo = System.currentTimeMillis() - 60*60*1000;
Filter query = QueryHelper.createFilter("loginTime > " + oneHourAgo
+ " or lastSeenTime > " + oneHourAgo);
int numActive = cache.keySet(query).size();
(See http://docs.oracle.com/cd/E15357_01/coh.360/e15723/api_cq.htm for more info on queries. All these examples were from Oracle Coherence.)
For the sake of full disclosure, I work at Oracle. The opinions and views expressed in this post are my own, and do not necessarily reflect the opinions or views of my employer.