I am trying to find a function in Matlab, or at least the name of an algorithm that does the following:
Let's say that I am analyzing a time series in real time. I initially start with a threshold of 10 and -10, so that when the time series goes above 10 or below -10, it's considered a 'HIT'. Let's say it initially takes the time series 5 minutes to produce a 'HIT', but I want to adjust the threshold so that, on average, it takes only 1 minute for a 'HIT' to be produced. I know it would look something like start with 10 and -10, if it takes too long, drop it to 5 and -5, then increase the threshold if it's too quick, etc.
I know there's a specific name for this type of algorithm, and there's probably built-in functions for this, but the name is eluding me. Can somebody help?
I don't know what the time resolution of your time series is, or if it's constant, so I'll leave that to you. However here is what you can do in matlab if you have a constant time resolution. First take the absolute value of the values in your time series. Then sort these values in reverse order using the sort() command. Then choose the value whose index in the sorted array gives you the average time resolution that you desire. So e.g. if your time series has size N and the time resolution is 0.1 seconds, and you want an alert on average every 1 second, then after sorting you would choose the threshold at (reverse order) sorted position N/10.
Related
There are multiple methods for find the average of a set of numbers.
First, the sum / count quotient. Add all values and divide them by the number of values.
Second, the moving average. The function I found in another Stack answer is:
New average = old average * (n-1)/n + new value /n
This works as long as each value is added to the average one value at a time.
My concern is that the second method is more calculatively complex for my processor to execute, but I also fear that the first method will result in a loss of resolution for data sets that result in large sums. In a 32 bit system, for example, the resolution of a float value stored is reduced automatically as the magnitude of the number grows.
Does a moving average preserve resolution?
"moving average" does not calculate average over large interval.
It smooth data in such way that newer measurements give larger impact, and older measurements weight becomes smaller and smaller.
If you bother about large sums and want to preserve all possible data "bits", consider special methods like Kahan summation algorithm
If I am given the total number of occurrences of an event over the last hour, and I can get this data at arbitrary times ( but at least once an hour ), how can I work out the total number of occurrences over a 24 hour period?
Obviously, you can't. For example -- if the first two observations overlap then it would be impossible to determine the number of kills during the overlap. If there is a time gap between the first two observations then there is no way to determine what happened during the gap. You could try to set up a system of equations -- but the resulting system will be underdetermined (but it could give you both a min and a max, which might be relevant).
Why not adopt a statistical approach? Let X = kills over a 1 hour period. This is a random variable. Estimate its expected value by sampling it at randomly chosen times and multiply your estimate by 24.
I recently posed a question on how to implement an algorithm which adaptively changes a threshold in real time, so that a time series reaches that threshold every N seconds. I was told that if my time series has a constant time interval (it does), I could take the absolute value, reverse sort it, and find the index in the array that gives me the average time resolution I want. I implemented it in MATLAB in this manner:
x = abs(timeseries); % Get the absolute value
x = flipud(sort(x)); % Reverse sort
N = length(x); % Size of the time series
idx = round(N/goal_time); % Find the right index
threshold = x(idx); % Set the threshold
Where goal time is the average time I want to detect a 'hit' (timeseries > threshold). My problem is two fold. I can't tell if the algorithm is not accurate enough, or if my data is too noisy (when using it, the average time I get hits doesn't really approximate my goal very well). And secondly, how would I modify this algorith to calculate a 'hit' time, where a hit is defined as the time series dipping below a threshold?
I am working on an algorithm to compute multiple mean max values of an array. The array contains time/value pairs such as HR data recorded on a Garmin device over a 5 hour run. the data is approx once a second for an unknown period, but has no guaranteed frequency. An example would be a 10 minute mean maximum, which is the maximum average 10 minute duration value. Assume "mean" is just average value for this discussion. The desired mean maximal value's duration is arbitrary, 1 min, 5 min, 60 min. And, I'm likely going to need many of them-- at least 30 but ideally any on demand if it wasn't a lengthy request.
Right now I have a straight forward algorithm to compute on value:
1) Start at beginning of array and "walk" forward until the subset is equal to or 1 element past the desired duration. Stop if end of array is reached.
2) Find the average of those subset values. Store as max avg if larger than current max.
3) Shift a single value off the left side of array.
4) Repeat from 1 until end of array met.
It basically computes every possible consecutive average and returns the max.
It does this for each duration. And it computes a real avg computation continuously instead of sliding it somehow by removing the left point and adding the right, like one could do for a Simple-moving-average series. It takes about 3-10 secs per mean max value depending on the total array size.
I'm wondering how to optimize this. For instance, the series of all mean max values will be an exponential curve with the 1s value highest, and lowering until the entire average is met. Can this curve, and all values, be interpolated from a certain number of points? Or some other optimization to the above heavy computation but still maintain accuracy?
"And it computes a real avg computation continuously instead of sliding it somehow by removing the left point and adding the right, like one could do for a Simple-moving-average series."
Why don't you just slide it (i.e. keep a running sum and divide by the number of elements in that sum)?
I'd like to sum up moving averages for a number of different categories when storing log records. Imagine a service that saves web server logs one entry at a time. Let's further imagine, we don't have access to the logged records. So we see them once but don't have access to them later on.
For different pages, I'd like to know
the total number of hits (easy)
a "recent" average (like one month or so)
a "long term" average (over a year)
Is there any clever algorithm/data model that allows to save such moving averages without having to recalculate them by summing up huge quantities of data?
I don't need an exact average (exactly 30 days or so) but just trend indicators. So some fuzziness is not a problem at all. It should just make sure that newer entries are weighted higher than older ones.
One solution probably would be to auto-create statistics records for each month. However, I don't even need past month statistics, so this seems like overkill. And it wouldn't give me a moving average but rather swap to new values from month to month.
An easy solution would be to keep an exponentially decaying total.
It can be calculated using the following formula:
newX = oldX * (p ^ (newT - oldT)) + delta
where oldX is the old value of your total (at time oldT), newX is the new value of your total (at time newT); delta is the contribution of new events to the total (for example the number of hits today); p is less or equal to 1 and is the decay factor. If we take p = 1, then we have the total number of hits. By decreasing p, we effectively decrease the interval our total describes.
If all you really want is a smoothed value with a given time constant then the easiest thing is to use a single pole recursive IIR filter (aka AR or auto-regressive filter in time series analysis). This takes the form:
Xnew = k * X_old + (1 - k) * x
where X_old is the previous smoothed value, X_new is the new smoothed value, x is the current data point and k is a factor which determines the time constant (usually a small value, < 0.1). You may need to determine the two k values (one value for "recent" and a smaller value for "long term") empirically, based on your sample rate, which ideally should be reasonably constant, e.g. one update per day.
It may be solution for you.
You can aggregate data to intermediate storage grouped by hour or day. Than grouping function will work very fast, because you will need to group small amount of records and inserts will be fast as well. Precision decisions up to you.
It can be better than auto-correlated exponential algorithms because you can understand what you calculate easier and it doesn't require math each step.
For last term data you can use capped collections with limited amount of records. They supported natively by some DBs for example MongoDB.