Suppose there is a real time feed of stock prices, how do you calculate the average of a subset of it (say over the past week)?
This was an interview question. I can come up with an algorithm to do it in O(n^2), but the interviewer wanted an algorithm that was O(n).
A useful approach is to compute the cumulative sum of your array.
This means that each entry in the cumulative sum array is the sum of all previous prices.
This is useful because you can then generate the sum over any particular subarray of your input using a single subtraction.
Note that when a new input arrives, you only need 1 addition to compute the new cumulative sum (because you simply add the new element to the old cumulative sum).
Another approach is akin to computing skew in Genomics.
If you want to compute the average over the past week, create a variable that contains the sum over a moving window. When an entry is created, add the entry to the above sum variable, and subtract the oldest entry in the moving window from it. Since the size of the window is constant, the average over the past week is just the moving sum over the number of entries in the past week.
Related
I'm sorry if this is the wrong place for this query. If it is, perhaps someone could direct me to the right place.
I have a program that has a bunch of objects (say n) to process and a process that iteratively processes one object.
At each iteration I have one less objects processed. I want to check if I need more objects.
If there are 100 objects or more, I have plenty. When there are less than 100 objects, say, I would like to get some more objects at a probability (P) that is roughly zero at 100 and 1 at 0 objects.
P(n) = 1 - (n/100)
If I just do a random calculation based on this probability then over time I get a cumulative probability that is the product of the series of probabilities which is not the same as the formula above.
If the probability added each time, I would get an integral of P(n), but since it is an accumulating product, what is the new function and how to calculate the function?
So I would like the total probability up till now to equal that formula. How do I work out the probability I need at the current iteration?
I realised after some thought that the answer is a simple integral, because the probability at each step is not independent, if I get more objects, the probability resets, if I don't get more objects the probability is the sum of all the times before that I didn't get more objects.
This is like a stock marketing problem, I am confused that the question is asking about how to get everyday's maximum profit? I only know that the algorithm's time complexity can be O(n) or O(n log2 n).
The input is A, an array of stock prices. For day i, the best trade is the maximum profit that can be achieved by buying at day i and selling on a subsequent day. For convenience, you can define the best trade for the last day to simply be −A[n] (because if you buy on the last day, you cannot sell and you lost all your money).
Give the pseudocode of an algorithm that returns an array containing the maximum profit for every day in A.
Update: I understand how to get the maximum profit now, and I can use the similar algorithm as the merge sort that divide and conquer to find this maximum profit. My question is what's another method (algorithm) that use time complexity O(n) to find the maximum profit or how can I approach in this way?
One way you can think of this problem is using one for loop since it's O(n), I can give you some hints:
for i from 0 to n:
if (A[i] < A[min]) // find the minimum value of stock
min = i;
profit = A[i] - A[min] // get the profit
if (profit > maxProfit) { // compares the profits
maxProfit = profit // always update the max profit
If you buy on day i, the maximum profit you can make is Amax(i) - A[i],
where Amax(i) is the highest price that occurs after day i.
My reading of the algorithm's specification is that you are to construct and return the array M whose entries are defined by M[i] = Amax(i) - A[i].
The highest price that occurs after day i is the greater of
A[i+i] and the highest price that occurs after day i + 1.
The last paragraph gives you a recursive relationship, except that unlike
the "typical" recursion you might see, the ith value depends on the
i + 1st value rather than the other way around.
But fortunately for you, you already know that the nth value and
every later value is 0, that is, after day n you will get 0 for your stock.
So you only need to figure out the values for days
1, 2, ..., n - 1, which you can do in O(n) time.
And each time you find one of these values, Amax(i), you can set one of the entries of M using M[i] = Amax(i) - A[i].
If you want to find the maximum profit that can be made by buying one share of stock on any single day and selling on any subsequent day, one time
(though this is not required by the problem statement, as far as I can see), you just have to find the maximum value in M, which you can do in O(n) time.
If your goal is to make the maximum profit possible by a series of actions, buying and selling stock, the best you can do is to buy as many shares as you can afford whenever the stock is at a local minimum (assuming infinite price before the first day) and sell everything whenever it reaches a local maximum price. You can identify all the local minimums and maximums in O(n) time, scanning A in either direction, and given a starting amount of money, you can compute the maximum overall profit in O(n) time using the list of local minimums and maximums of A. (But this does not use the array that the original problem statement asked you to construct, because that array does not account for the number of shares bought nor the possibility of multiple transactions.)
Remember that if each pass of a two-pass algorithm takes O(n) time, the algorithm as a whole takes O(n) time.
I am working on an algorithm to compute multiple mean max values of an array. The array contains time/value pairs such as HR data recorded on a Garmin device over a 5 hour run. the data is approx once a second for an unknown period, but has no guaranteed frequency. An example would be a 10 minute mean maximum, which is the maximum average 10 minute duration value. Assume "mean" is just average value for this discussion. The desired mean maximal value's duration is arbitrary, 1 min, 5 min, 60 min. And, I'm likely going to need many of them-- at least 30 but ideally any on demand if it wasn't a lengthy request.
Right now I have a straight forward algorithm to compute on value:
1) Start at beginning of array and "walk" forward until the subset is equal to or 1 element past the desired duration. Stop if end of array is reached.
2) Find the average of those subset values. Store as max avg if larger than current max.
3) Shift a single value off the left side of array.
4) Repeat from 1 until end of array met.
It basically computes every possible consecutive average and returns the max.
It does this for each duration. And it computes a real avg computation continuously instead of sliding it somehow by removing the left point and adding the right, like one could do for a Simple-moving-average series. It takes about 3-10 secs per mean max value depending on the total array size.
I'm wondering how to optimize this. For instance, the series of all mean max values will be an exponential curve with the 1s value highest, and lowering until the entire average is met. Can this curve, and all values, be interpolated from a certain number of points? Or some other optimization to the above heavy computation but still maintain accuracy?
"And it computes a real avg computation continuously instead of sliding it somehow by removing the left point and adding the right, like one could do for a Simple-moving-average series."
Why don't you just slide it (i.e. keep a running sum and divide by the number of elements in that sum)?
I'd like to sum up moving averages for a number of different categories when storing log records. Imagine a service that saves web server logs one entry at a time. Let's further imagine, we don't have access to the logged records. So we see them once but don't have access to them later on.
For different pages, I'd like to know
the total number of hits (easy)
a "recent" average (like one month or so)
a "long term" average (over a year)
Is there any clever algorithm/data model that allows to save such moving averages without having to recalculate them by summing up huge quantities of data?
I don't need an exact average (exactly 30 days or so) but just trend indicators. So some fuzziness is not a problem at all. It should just make sure that newer entries are weighted higher than older ones.
One solution probably would be to auto-create statistics records for each month. However, I don't even need past month statistics, so this seems like overkill. And it wouldn't give me a moving average but rather swap to new values from month to month.
An easy solution would be to keep an exponentially decaying total.
It can be calculated using the following formula:
newX = oldX * (p ^ (newT - oldT)) + delta
where oldX is the old value of your total (at time oldT), newX is the new value of your total (at time newT); delta is the contribution of new events to the total (for example the number of hits today); p is less or equal to 1 and is the decay factor. If we take p = 1, then we have the total number of hits. By decreasing p, we effectively decrease the interval our total describes.
If all you really want is a smoothed value with a given time constant then the easiest thing is to use a single pole recursive IIR filter (aka AR or auto-regressive filter in time series analysis). This takes the form:
Xnew = k * X_old + (1 - k) * x
where X_old is the previous smoothed value, X_new is the new smoothed value, x is the current data point and k is a factor which determines the time constant (usually a small value, < 0.1). You may need to determine the two k values (one value for "recent" and a smaller value for "long term") empirically, based on your sample rate, which ideally should be reasonably constant, e.g. one update per day.
It may be solution for you.
You can aggregate data to intermediate storage grouped by hour or day. Than grouping function will work very fast, because you will need to group small amount of records and inserts will be fast as well. Precision decisions up to you.
It can be better than auto-correlated exponential algorithms because you can understand what you calculate easier and it doesn't require math each step.
For last term data you can use capped collections with limited amount of records. They supported natively by some DBs for example MongoDB.
I have a process that generates values and that I observe. When the process terminates, I want to compute the median of those values.
If I had to compute the mean, I could just store the sum and the number of generated values and thus have O(1) memory requirement. How about the median? Is there a way to save on the obvious O(n) coming from storing all the values?
Edit: Interested in 2 cases: 1) the stream length is known, 2) it's not.
You are going to need to store at least ceil(n/2) points, because any one of the first n/2 points could be the median. It is probably simplest to just store the points and find the median. If saving ceil(n/2) points is of value, then read in the first n/2 points into a sorted list (a binary tree is probably best), then as new points are added throw out the low or high points and keep track of the number of points on either end thrown out.
Edit:
If the stream length is unknown, then obviously, as Stephen observed in the comments, then we have no choice but to remember everything. If duplicate items are likely, we could possibly save a bit of memory using Dolphins idea of storing values and counts.
I had the same problem and got a way that has not been posted here. Hopefully my answer can help someone in the future.
If you know your value range and don't care much about median value precision, you can incrementally create a histogram of quantized values using constant memory. Then it is easy to find median or any position of values, with your quantization error.
For example, suppose your data stream is image pixel values and you know these values are integers all falling within 0~255. To create the image histogram incrementally, just create 256 counters (bins) starting from zeros and count one on the bin corresponding to the pixel value while scanning through the input. Once the histogram is created, find the first cumulative count that is larger than half of the data size to get median.
For data that are real numbers, you can still compute histogram with each bin having quantized values (e.g. bins of 10's, 1's, or 0.1's etc.), depending on your expected data value range and precision you want.
If you don't know the value range of entire data sample, you can still estimate the possible value range of median and compute histogram within this range. This drops outliers by nature but is exactly what we want when computing median.
You can
Use statistics, if that's acceptable - for example, you could use sampling.
Use knowledge about your number stream
using a counting sort like approach: k distinct values means storing O(k) memory)
or toss out known outliers and keep a (high,low) counter.
If you know you have no duplicates, you could use a bitmap... but that's just a smaller constant for O(n).
If you have discrete values and lots of repetition you could store the values and counts, which would save a bit of space.
Possibly at stages through the computation you could discard the top 'n' and bottom 'n' values, as long as you are sure that the median is not in that top or bottom range.
e.g. Let's say you are expecting 100,000 values. Every time your stored number gets to (say) 12,000 you could discard the highest 1000 and lowest 1000, dropping storage back to 10,000.
If the distribution of values is fairly consistent, this would work well. However if there is a possibility that you will receive a large number of very high or very low values near the end, that might distort your computation. Basically if you discard a "high" value that is less than the (eventual) median or a "low" value that is equal or greater than the (eventual) median then your calculation is off.
Update
Bit of an example
Let's say that the data set is the numbers 1,2,3,4,5,6,7,8,9.
By inspection the median is 5.
Let's say that the first 5 numbers you get are 1,3,5,7,9.
To save space we discard the highest and lowest, leaving 3,5,7
Now get two more, 2,6 so our storage is 2,3,5,6,7
Discard the highest and lowest, leaving 3,5,6
Get the last two 4,8 and we have 3,4,5,6,8
Median is still 5 and the world is a good place.
However, lets say that the first five numbers we get are 1,2,3,4,5
Discard top and bottom leaving 2,3,4
Get two more 6,7 and we have 2,3,4,6,7
Discard top and bottom leaving 3,4,6
Get last two 8,9 and we have 3,4,6,8,9
With a median of 6 which is incorrect.
If our numbers are well distributed, we can keep trimming the extremities. If they might be bunched in lots of large or lots of small numbers, then discarding is risky.