I have a sensor that gives me some value every x minutes. In my database I store the values grouped by hour. So the input domain is an integer between 0 and 23. I want to learn the parameters for a Gaussian distribution (mean and variance). At the end I want to get the probability of a certain measurement level at any given time (hour of day).
The problem is that the peak of the distribution could be at midnight and in the afternoon there could be the minimum. In this case I have no idea how to model it with a Gaussian as the normal distribution does not seem to be suited for "circular" domains (after 23h comes 0h and not 24h, so it is not really continuous).
How could this be modeled? Can I use a normal Gaussian here or should I go for another kind of probability model?
Example:
Here I would expect the model to have mean = 23 or 0. But I guess when I learn parameters here, the mean would be exactly in the middle, because a Gaussian only has one peak.
Related
My friends and I are competing in our own fitness challenge (Sober October) where we are keeping track of Activity, Total Time Spent Moving, and Distance. Our activities include running (outdoors), running (treadmill), running (elliptical), rowing, biking (stationary), biking (outdoors), swimming, and stair stepper.
As a group, we weren't really interested in using a calorie estimation because those results can be easily manipulated by increasing the weight that the equation uses, so we wanted to keep it based on just distance and time.
What kind of equation should I use to best normalize such exercises? I'm looking for something that would weight distance and time differently based on the activity; for example, when compared to running,biking should give more weight to time than to milage because it takes less work to go a mile on a bike than it does on foot.
I was able to find this article on how calories are calculated, and just thought about removing the weight portion of the equation to get our normalized number, but wanted to see if there was a better way to calculate what I'm looking for.
Objective measure
You are seeking an objective measurement which is independent of weight. Use METs.
A human expends a baseline of one MET sitting quietly. Maybe your measure will be excess-MET-hours.
Score = (METs - 1) × Hours
MET values
On that link above you can find reference METs values for various activities, including several of your target activities. These are independent of speed.
You can further improve the calculation by factoring in your distance/time measurements. For example, given cited METs figures:
Walking slowly (1 mph) = 2.0 MET
Walking (3 mph) = 3.0 MET
Jogging (6.8 mph) = 11.2 MET
You can fit them to a curve. Use Desmos.
So your score for walking/jogging/running is:
Excess METs = [1 + 0.2 × (miles/hours) ^ 2 - 1] × hours
You can make similar estimations for other activities.
I'm logging temperature values in a room, saving them to the database. I'd like to be alerted when temperature rises suddenly. I can't set fixed values, because 18°C is acceptable in winter and 25°C is acceptable in summer. But if it jumps from 20°C to 25°C during, let's say, 30 minutes and stays like this for 5 minutes (to eliminate false readouts), I'd like to be informed.
My current idea is to take readouts from last 30 minutes (A) and readouts from last 5 minutes (B), calculate median of A and B and check if difference between them is less then my desired threshold.
Is this correct way to solve this or is there a better algorithm? I searched for a specific one but most of them seem overcomplicated.
Thanks!
Detecting changes in a time-series is a well-researched subject, and hundreds if not thousands of papers have been written on this subject. As you've seen many methods are quite advanced, but proved to be quite useful for many use cases. Whatever method you choose, you should evaluate it against real of simulated data, and optimize its parameters for your use case.
As you require, let me suggest a very simple method that in many cases prove to be good enough, and is quite similar to that you considered.
Basically, you have two concerns:
Detecting a monotonous change in a sampled noisy signal
Ignoring false readouts
First, note that medians are not commonly used for detecting trends. For the series (1,2,3,30,35,3,2,1) the medians of 5 consecutive terms is be (3, 3, 3, 3). It is much more common to use averages.
One common trick is to throw the extreme values before averaging (e.g. for each 7 values average only the middle 5). If many false readouts are expected - try to take measurements at a faster rate, and throw more extreme values (e.g. for each 13 values average the middle 9).
Also, you should throw away unfeasible values and replace them with the last measured value (unfeasible means out of range, or non-physical change rate).
Your idea of comparing a short-period measure with a long-period measure is a good idea, and indeed it is commonly used (e.g. in econometrics).
Quoting from "Financial Econometric Models - Some Contributions to the Field [Nicolau, 2007]:
Buy and sell signals are generated by two moving averages of the price
level: a long-period average and a short-period average. A typical
moving average trading rule prescribes a buy (sell) when the
short-period moving average crosses the long-period moving average
from below (above) (i.e. when the original time series is rising
(falling) relatively fast).
When you say "rises suddenly," mathematically you are talking about the magnitude of the derivative of the temperature signal.
There is a nice algorithm to simultaneously smooth a signal and calculate its derivative called the Savitzky–Golay filter. It's explained with examples on Wikipedia, or you can use Matlab to help you generate the convolution coefficients required. Once you have the coefficients the calculation is very simple.
I'm creating an app to monitor water quality. The temperature data is updated every 2 min to firebase real-time database. App has two requirements
1) It should alert the user when temperature exceed 33 degree or drop below 23 degree - This part is done
2) It should alert user when it has big temperature fluctuation after analysing data every 30min - This part i'm confused.
I don't know what algorithm to use to detect big temperature fluctuation over a period of time and alert the user. Can someone help me on this?
For a period of 30 minutes, your app would give you 15 values.
If you want to figure out a big change in this data, then there is one way to do so.
You can use implement the following method:
Calculate the mean and the standard deviation of the values.
Subtract the data you have from the mean and then take the absolute value of the result.
Compare if the absolute value is greater than one standard deviation, if it is greater then you have a big data.
See this example for better understanding:
Lets suppose you have these values for 10 minutes:
25,27,24,35,28
First Step:
Mean = 27 (apprx)
One standard deviation = 3.8
Second Step: Absolute(Data - Mean)
abs(25-27) = 2
abs(27-27) = 0
abs(24-27) = 3
abs(35-27) = 8
abs(28-27) = 1
Third Step
Check if any of the subtraction is greater than standard deviation
abs(35-27) gives 8 which is greater than 3.8
So, there is a big fluctuation. If all the subtracted results are less than standard deviation, then there is no fluctuation.
You can still improvise the result by selecting two or three standard deviation instead of one standard deviation.
Start by defining what you mean by fluctuation.
You don't say what temperature scale you're using. Fahrenheit, Celsius, Rankine, or Kelvin?
Your sampling rate is a new data value every two minutes. Do you define fluctuation as the absolute value of the difference between the last point and current value? That's defensible.
If the max allowable absolute value is some multiple of your 33-23 = 10 degrees you're in business.
A hardware sensor is sampled precisely (precise period of sampling) using a real-time unit. However, the time value is not sent to the database together with the sampled value. Instead, time of insertion of the record to the database is stored for the sample in the database. The DATETIME type is used, and the GETDATE() function is used to get current time (Microsoft SQL Server).
How can I reconstruct the precise sampling times?
As the sampling interval is (should be) 60 seconds exactly, there was no need earlier for more precise solution. (This is an old solution, third party, with a lot of historical samples. This way it is not possible to fix the design.)
For processing of the samples, I need to reconstruct the correct time instances for the samples. There is no problem with shifting the time of the whole sequence (that is, it does not matter whether the start time is rather off, not absolute). On the other hand, the sampling interval should be detected as precisely as possible. I also cannot be sure, that the sampling interval was exactly 60 seconds (as mentioned above). I also cannot be sure, that the sampling interval was really constant (say, slight differences based on temperature of the device).
When processing the samples, I want to get:
start time
the sampling interval
the sequence o the sample values
When reconstructing the samples, I need to convert it back to tuples:
time of the sample
value of the sample
Because of that, for the sequence with n samples, the time of the last sample should be equal to start_time + sampling_interval * (n - 1), and it should be reasonably close to the original end time stored in the database.
Think in terms of the stored sample times slightly oscillate with respect to the real sample-times (the constant delay between the sampling and the insertion into the database is not a problem here).
I was thinking about calculating the mean value and the corrected standard deviation for the interval calculated from the previous and current sample times.
Discontinuity detection: If the calculated interval is greater than 3 sigma off the mean value, I would consider it a discontinuity of the sampled curve (say, the machine is switched off, or any outer event lead to missing samples. In the case, I want to start with processing a new sequence. (The sampling frequency could also be changed.)
Is there any well known approach to the problem. If yes, can you point me to the article(s)? Or can you give me the name or acronym of the algorithm?
+1 to looking at the difference sequence. We can model the difference sequence as the sum of a low frequency truth (the true rate of the samples, slowly varying over time) and high frequency noise (the random delay to get the sample into the database). You want a low-pass filter to remove the latter.
I need to calculate the nearest dataPoint in a time series chart from a specific point in a chart
I obviously cannot use d=sqrt(x*x+y*y) as my x axis is in time series, hence it wont make sense to have an equation where I am adding distance and time together (x,y need to have same units). Moreover visually it may seem right, but it still depends upon the scale of the x axis.
So what best logic can I use to find the nearest point?
I can think of using a quadratic form of x (i.e. time) so as that my final function can ne f(x*x,y), but then it is just a subjective equation.
Does anyone have a better and more logical approach to this. If there is an intuitive logical approach I will love it. And if there is a complicated model I would still like to know about it and explore it.
Thanks
EDIT
TO give background: I am polling people to predict where the stock price will be in April(they have to mention exact date when the expect price to be there) ... How do I measure their performance?
One intuitive way is by calculating the average absolute change per day.
i.e.
Sum of Absolute changes every day from previous day / Total number of days in series.
Thereafter I can translate each day in terms of prices i.e. the average price change per day.
Thus if average absolute change per day is lets say 2, then a price that is 10 days away can be said to be 20 price points away.
Thereafter I can calculate the distance based on sqrt(x*x+y*y) formula.
This can be fine tuned by using a bell curve (std dev and mean) rather than just mean of absolute change per day. But then it will make solution more ocmplicated.