Related
I am making a BI system for a bank-like institution. This system should manage credit contracts, invoices, payments, penalties and interest.
Now, I need to make a method that builds an invoice. I have to calculate how much the customer has to pay right now. He has a debt, which he has to pay for. He also has to pay for the interest. If he was ever late with due payment, penalties are applied for each day he's late.
I thought there were 2 ways of doing this:
By having only 1 original state - the contract's original state. And each time to compute the monthly payment which the customer has to make, consider the actual, made payments.
By constantly making intermediary states, going from the last intermediary state, and considering only the events that took place between the time of these 2 intermediary states. This means having a job that performs periodically (daily, monthly), that takes the last saved state, apply the changes (due payments, actual payments, changes in global constans like the penalty rate which is controlled by the Central Bank), and save the resulting state.
The benefits of the first variant:
Always actual. If changes were made with a date from the past (a guy came with a paid invoice 5 days after he made the payment to the bank), they will be correctly reflected in the results.
The flaws of the first variant:
Takes long to compute
Documents printed with the current results may differ if the correct data changes due to operations entered with a back date.
The benefits of the second variant:
Works fast, and aggregated data is always available for search and reports.
Simpler to compute
The flaws of the second variant:
Vulnerable to failed jobs.
Errors in the past propagate until the end, to the final results.
An intermediary result cannot be changed if new data from past transactions arrives (it can, but it's hard, and with many implications, so I'd rather mark it as Tabu)
Jobs cannot be performed successfully and without problems if an unfinished transaction exists (an issued invoice that wasn't yet paid)
Is there any other way? Can I combine the benefits from these two? Which one is used in other similar systems you've encountered? Please share any experience.
Problems of this nature are always more complicated than they first appear. This
is a consequence of what I like to call the Rumsfeldian problem of the unknown unknown.
Basically, whatever you do now, be prepared to make adjustments for arbitrary future rules.
This is a tough proposition. some future possibilities that may have a significant impact on
your calculation model are back dated payments, adjustments and charges.
Forgiven interest periods may also become an issue (particularly if back dated). Requirements
to provide various point-in-time (PIT) calculations based on either what was "known" at
that PIT (past view of the past) or taking into account transactions occurring after the reference PIT that
were back dated to a PIT before the reference (current view of the past). Calculations of this nature can be
a real pain in the head.
My advice would be to calculate from "scratch" (ie. first variant). Implement optimizations (eg. second variant) only
when necessary to meet performance constraints. Doing calculations from the beginning is a compute intensive
model but is generally more flexible with respect to accommodating unexpected left turns.
If performance is a problem but the frequency of complicating factors (eg. back dated transactions)
is relatively low you could explore a hybrid model employing the best of both variants. Here you store the
current state and calculate forward
using only those transactions that posted since the last stored state to create a new current state. If you hit a
"complication" re-do the entire account from the
beginning to reestablish the current state.
Being able to accommodate the unexpected without triggering a re-write is probably more important in the long run
than shaving calculation time right now. Do not place restrictions on your computation model until you have to. Saving
current state often brings with it a number of built in assumptions and restrictions that reduce wiggle room for
accommodating future requirements.
Occasionally I read news articles that mention about some computer models that computer scientists use to predict winners of some sporting events or the odds for betting which I think there must be a mathematical model behind it. I never bothered to think twice even though I am a "pseudo computer scientist" myself. With the 2010 FIFA World Cup just underway, and since I am also a "pseudo football/soccer player" myself, I just started to wonder about these calculations algorithms.
For example, I know one factor is determining the strength of opponents, so that a win against a strong opponent can count more than a win against a weak opponent. But it now kind of gets in a circular loop, or at least how does one determine the strength of a team in the first place, before that team can be considered strong or weak? If it's based on a historical data then there's no way that could be accurate, because those players of the past are no longer on the fields so their impact is none (except maybe if they become coaches like Maradona)
Anyway, long question short, if you're happen to be working in this field or have some knowledge, please shed some lights.
I know of some work, but its basis might surprise you a bit -- it's been used to predict (quite accurately) what countries do how well in the Olympics, but it's based purely on the economics of the countries in question, not looking at the individual athletes at all. I don't believe it's been used specifically to look at FIFA world cup, but I suspect it would apply about as well, or maybe even a bit better.
Some of the large Investment Banks started a competition for thier quants to write models to predict the wold cup winner.
http://kaggle.com/worldcup2010
More info on the models
http://kaggle.com/blog/2010/06/03/predicting-the-2010-fifa-world-cup-can-statisticians-outdo-the-investment-banks/
There's been some modeling to select horse racing winners using logit models here. The general principle can be applied to predicting which teams advance to the Round of 16 and subsequent rounds. Horse racing is at least, if not more, complex with regard to the number of variables that have a statistically significant effect on the outcome. For instance, in the author's model, weight, win rate, jockey characteristics, speed, post position, distance, and winnings were all significant variables. The authors didn't have access to "trip handicapping" at the time which has proven to be another important effect.
Reading this paper might help generate some thoughts around handicapping FIFA.
To tanascius point, developing a predictive model is the first step. As the authors further explain, developing a betting strategy based on the results is a different problem that's based, in part, on the accuracy of the model.
One guy has been using googles page-rank for sports. Not sure why he felt the need to rename it:
http://www.physorg.com/news180094320.html
I found that like by a quick search for using page-rank for sports because I realized it solves the circular references in rankings. Was curious if anyone had tried it, and there it was.
BTW, anyone who can make accurate predictions for things you can bet on is not publishing their methods or results. They should be making money instead.
I think there's too much to take into consideration:
Injured players, form players, form teams, pressure on teams, rivalries, weather, home advantage, past meetings, formations, team styles, expectations....
Dunno if this applies to the FIFA soccer video games, but I know that for the Super Bowl (american football, for those who dont know) they use the latest version of Madden to predict the winner.
Not very scientific, I suppose, but its there.
A delay will always occur between a user action and an application response.
It is well known that the lower the response delay, the greater the feeling of the application responding instantaneously. It is also commonly known that a delay of up to 100ms is generally not perceivable. But what about a delay of 110ms?
What is the shortest application response delay that can be perceived?
I'm interested in any solid evidence, general thoughts and opinions.
The 100 ms threshold was established over 30 yrs ago. See:
Card, S. K., Robertson, G. G., and Mackinlay, J. D. (1991). The information visualizer: An information workspace. Proc. ACM CHI'91 Conf. (New Orleans, LA, 28 April-2 May), 181-188.
Miller, R. B. (1968). Response time in man-computer conversational transactions. Proc. AFIPS Fall Joint Computer Conference Vol. 33, 267-277.
Myers, B. A. (1985). The importance of percent-done progress indicators for computer-human interfaces. Proc. ACM CHI'85 Conf. (San Francisco, CA, 14-18 April), 11-17.
What I remember learning was that any latency of more than 1/10th of a second (100ms) for the appearance of letters after typing them begins to negatively impact productivity (you instinctively slow down, less sure you have typed correctly, for example), but that below that level of latency productivity is essentially flat.
Given that description, it's possible that a latency of less than 100ms might be perceivable as not being instantaneous (for example, trained baseball umpires can probably resolve the order of two events even closer together than 100ms), but it is fast enough to be considered an immediate response for feedback, as far as effects on productivity. A latency of 100ms and greater is definitely perceivable, even if it's still reasonably fast.
That's for visual feedback that a specific input has been received. Then there'd be a standard of responsiveness in a requested operation. If you click on a form button, getting visual feedback of that click (eg. the button displays a "depressed" look) within 100ms is still ideal, but after that you expect something else to happen. If nothing happens within a second or two, as others have said, you really wonder if it took the click or ignored it, thus the standard of displaying some sort of "working..." indicator when an operation might take more than a second before showing a clear effect (eg. waiting for a new window to pop up).
New research as of January, 2014:
http://newsoffice.mit.edu/2014/in-the-blink-of-an-eye-0116
...a team of neuroscientists from MIT has found that the human brain
can process entire images that the eye sees for as little as 13
milliseconds...That speed is far faster than the 100 milliseconds
suggested by previous studies...
At the San Francisco Opera house, we routinely setup precise delay setting for each of our speakers. We can detect 5 millisecond changes in delay times to our speakers. When you make such subtle changes, you change where the sound sources from. Often times we want sound to sound as if it's coming from someplace other than were the speakers are. Precise delay adjustments make this possible. Sound delays of 15 milliseconds are very obvious even to untrained ears because it radically shifts where the sound sources from. A simple test is to prove this is to play sound through multiple speakers, and have the subject close their eyes and point to where the sound is coming from. Now make a slight change in the delay time to one of the speakers of just a few milliseconds, and have the person point again to where the sound is coming from. Making changes in delay times is acoustically very similar to moving the actual speakers.
I don't think anecdotes or opinions are really valid for answers here. This question touches on the psychology of user experience and the sub-conscious mind. The human brain is powerful and fast and mere milliseconds do count and are registered. I am no expert but I know there is much science behind e.g. what Matt Jacobsen mentioned. Check out Google's study here http://services.google.com/fh/files/blogs/google_delayexp.pdf for an idea of how much it can affect site traffic.
Here's another study by Akami - 2 second response time
http://www.akamai.com/html/about/press/releases/2009/press_091409.html (From https://ux.stackexchange.com/questions/5529/once-apon-a-time-there-was-a-10-seconds-to-load-a-page-rule-what-is-it-nowa )
Does anyone have any other studies to share?
Persistence of vision is around 100ms so it should be a reasonable visual feedback delay. 110ms should make no difference, as it is an approximate value. In practice you won't notice a delay below 200ms.
Out of my memory, studies have shown that users lose patience and retry an operation after around 2s of inactivity (in the absence of feedback), e.g. clicking on a confirm or action button. So plan on using some kind of animation if the action takes longer than 1s.
I worked on an application that had a explicit business goal of being blindingly fast, and we had a max allowed server time of 150ms for processing a full web page.
No solid evidence but for our own application, we allow a maximum of one second between a user action and feedback. If it does take longer, a "waiting box" should be shown.
A user should see "something" happening within a second of causing an action.
Use the dual of test for visual spatial resolution ( two parallel black bars, with an equal width, and an equal gap between them. Reduce angular subtense until they appear to be one line, ie scale down or simply move away. The point at which it seems to merge into one line shows the threshold).
Use function gen to blink an LED on for an interval, then off, then on, then off --- same time delay each interval, but repeat the pattern while gradually decreasing that delay, thus same as above, but time in place of space.
Imagine an oscilloscope image like so:
_________/^d^\_d_/^d^\_________
I note that at 41 ms interval, I perceive one longer blink only, but at 42 ms, I just perceive it as extremely rapid double blink. Thus, threshold is ~42ms. Probably varies depending on person, age, condition etc.
This is close to 24 fps, which is probably why cinema works at that presentation rate.
Reaction time to see something, then decide to react, say by clicking mouse etc, is longer much longer again. Thus, it's not surprising that experiments requiring a reaction response to measure yield a longer time, but that longer delay wasn't what you were asking for, and the above experiment is easy and illuminating!
But note also -- smoothly moving animations require the visual cortex to work harder, delaying visual comprehension. This delay is 'hidden' from perception, so longer delays (several hundred ms) can be 'hidden' by just providing something thats difficult to see because moving.
The effect that hides it is called Chronostasis. Basically, glancing somewhere 'new' requires the visual cortex to work harder to 'de-render' / 'recognise' the scene. This takes a remarkably long time, during which your consciousness is essentially 'paused'.
Once looking at a mostly-constant scene, only changes need this processing, so smaller/faster changes are possible and your perceptual experience resumes, and faster/smaller movements are detectable.
The detection of changes visually is processed basically on your retina. Your eyes also have a natural 'bandpass' response -- stare unblinkingly at anything for sufficient time, and at sufficient distance for saccades to be unable to change the image much, and you will find your visual feed fading out to 'grey'. This is what gives us our 'white balance', and is somewhat similar to the automatic gain control on analogue radio/tv.
The point is, that your eyes themselves have a time constant to respond, but this is actually dependant on the strength of the stimulus. (brightness of the LED, for our case).
Too bright, and the ability of your retinal cells to 'relax' back from the brightness, ie, respond to the 'sudden dark', is compromised.
The effect which keeps you seeing bright things after the light has stopped is called 'persistence of vision', and old cathode-ray picture tubes more or less depend heavily on it for them to work at all.
This is the one that's usually 100 ms or so, but it's not a 'sharp' interval -- more like a exponential roll-off, and again -- changes duration depending on how bright the stimulus is relative to how dark-adjusted (ie, sensitive) the eye is at that moment.
For duller, faster changes, especially changes outside your fovea, you will perceive even higher rates easily. Eg, flickering lights. Those outer parts of your retina (most of the area, actually) are adapted to detecting movement, and bringing it to your attention. So it makes sense that although lacking spatial resolution, they have greater time resolution / shorter response rate.
But this also means animating things usually requires even finer time steps, otherwise 'jumpiness' is perceptible, mostly due to that faster response.
Note all the scaling/sliding full screen animations iOS uses -- these essentially exploit chronostasis to hide technically unavoidable loading delays, giving the perception that those products respond instantly and smoothly at all times.
So, show something different within 42 ms -> instant response.
Keep animating otherwise useless hard-to-see-properly visuals continuously at high frame rates, then stop suddenly when done -> hides the delay so long as enough is visually busy, and the delay isn't too long. (probably 250ms is pushing the friendship).
This also seems to tee up with other's perceptions of input lag, for example : http://danluu.com/input-lag/
100ms is totally wrong. You can prove this yourself using your fingers, a desk, and a watch with visible seconds. Synchronising to the watch's seconds, drum out beats on the desk continuously such that 16 beats are drummed out every second. I chose 16 because it is natural to drum out multiples of two, so it's like four strong beats with three weak beats in between. Adjacent beats are clearly discernible by their sound. The beats are separated by about 60ms, so even 60 ms is actually still too high. Therefore the threshold is way below 100ms, especially if sound is involved.
For instance, a drum app or a keyboard app needs a delay of more like 30ms, or else it gets really annoying, because you hear the sound coming from the physical button / pad / key well before the sound comes out of the speakers. Software like ASIO and jack were made specifically to deal with this issue, so no excuses. If your drum app has a 100ms delay, I will hate you.
The situation for VoIP and high powered gaming is actually worse, because you need to react to events in real time, and in music, at least you get to plan ahead at least a little. For an average human reaction time of 200ms, a further 100ms delay is an enormous penalty. It noticeably changes the conversational flow of VoIP. In gaming, 200ms reaction time is generous, especially if the players have a lot of practice.
For a reasonably current scholarly article, try out How Much Faster is Fast Enough? User Perception of
Latency & Latency Improvements in Direct and Indirect Touch (PDF). While the main focus was on JND (Just Noticeable Difference) of delay, there is some good background on on absolute delay perception and they also acknowledge and account for 60Hz monitors (16.7 ms repaint times) in their second experiment.
I am a cognitive neuroscientist who studies visual perception and cognition.
The paper by Mary Potter mentioned above regards the minimum time required to categorize a visual stimulus. However, understand that this is under laboratory conditions in the absence of any other visual stimuli, which certainly would not be the case in the real world user experience.
The typical benchmark for a stimulus-response / input-stimulus interaction, that is, the average amount of time for an individuals minimum reaction speed or input-response detection is ~200ms. to be certain there is no detectable difference, this threshold could be lowered to around 100ms. Below this threshold, the temporal dynamics of your cognitive processes take longer to compute the event than the event itself, so there is nearly no chance of any ability to detect or differentiate it. You could go lower to say 50 ms, but it really wouldn't be necessary. 10 ms and you've gone into the territory of overkill.
For web applications 200ms is considered as unnoticable delay, while 500ms is acceptable.
I'm writing an app that lets a diabetic user enter his/her "blood glucose" readings, and then charts them on a graph over time from left to right. Since the blood readings will be done only several times a day, an algorithm would be handy to:
a) fill in the gaps on the graph between readings (curves would be more realistic than jerky lines) and allow a more accurate "blood glucose level" daily average
b) roughly predict what will happen in the future (if the user eats nothing that will affect his blood levels)
I suck at calculus. I'm hoping someone here knows a library for this stuff? I'm hoping someone knows of an algorithm that has been tailored for this specific problem already (e.g.: where someone has compared it to real data from diabetics)
Disclaimer: I am very aware that any such algorithm would vary wildly depending on the user. I'm just looking to improve on straight angular lines. Regardless of the diabetic, there is a limit to the rate that blood sugars can rise and fall.
I'm using Javascript, but as it's just math, I could port it from C, Java or whatever.
Blood sugar behavior is very complicated. It is affected by
Current blood sugar (complicated by the possible presence of ketones if the patient is hyperglycemic)
recent food out to several hours depending on the type and how much
recent fast acting insulin (with variety and patient dependent reaction profiles between 45 minutes and two hours long. Oh, and delivery mechanism)
long-acting insulin out past 12 hours (again patient and variety dependent)
activity levels
stress levels
illness
basal insulin rate if the patient wears a pump
ad nauseum
Very hard problem. Any heuristic---any heuristic---you chose would be highly misleading. So short answer:
Don't do it.
This comes, in part, from having compared a diabetic's 24-hour continous glucose log with the ~10 finger pricks taken during the same time. I.e. my suggestion is data driven.
Edit: Evidently I didn't make myself clear.
You can't even get close.
Nothing you can do with finger prick data can be remotely reliable.
Connecting the dots with any lines (even straight segments) is just plain wrong. It doesn't reflect reality. Not even a little bit.
I'm an experimental particle physicist. Complicated data sets are what I do. There is a diabetic in my life (did you guess?). This matters to me.
But I've seen the high frequency data logs, side-by-side with a log of the days finger-pricks, exercise, food, and insulin.
If you could get every-fifteen-minutes data, I'd say go ahead and use a spline. It won't be dangerously misleading. But, if you have 6-10 measurements across the day, you know nothing.
Good news: continuous monitoring is coming down in price. It's out of the lab and available with some pumps even now.
For those who aren't familiar with this: compliant diabetic patients do (results of extremely unscientific polling) 4-6+ glucose tests a day as a matter of course, and several additional ones in the 1-2 hours following any unexpected excursion (they get physical symptoms that allow them to detect severe excursions).
This serves to give the patient a rough idea of how they are doing at controlling their glucose levels, but they also go to a lab to get a Hemoglobin A1C drawn every quarter (or so). The A1C result is dependent mostly on their average blood glucose.
I've talked to people who clocked in a 80-110 (quite favorable numbers) four times a day for months, and got back an A1C suggesting an average above 150 (not desirable at all). Presumable they were going high in the night. And I've heard similar stories from people who we probably going low---very low---in their sleep.
The lesson is:
Finger prick readings have their place, but don't try to extrapolate them to times not well sampled.
If you want to do just a straight fit of the data to make things easier to view then something like what Charlie Martin recommended would likely work well. However, as noted by dmckee this data really wouldn't mean anything.
What you are trying to do is actually more in line with pharmacokenetics which is an entire scientific study in and of itself. In this case I'm not even sure it would entirely apply except in the case of Type I Diabetes as most of what I know about pharamcokenetics only applies drug studies, but if something is being produced by the body then you are likely looking at entirely different types of analysis. If you are interested in the subject then there are quite a few book previews on Google Books if you do a search for "pharmacokienetics" but due to the nature of the subject they are very math heavy and assume that you have an understanding of chemistry and biology as well.
okay, you're going to be looking for some fitted curve. The thing with that is that for n points there are fit polynomials up to order... n-1 I think. It's been a while. Yep. by golly, I'm right. The common thing when you have lots of points and don't wants a complicated function (which you don't) is to use a least-squares approximation.
probably the best thing is to look for a canned routine you can use; these exist in most stats packages. Give us a little more detail on the environment you want and we might be able to point you more closely to something suitable.
This is most likely not going to work but Artificial Neural Networks may, and i repeat may be able to get something out of a good data set. By good, i mean like weeks or months of continuous recording, and even then i wouldn't trust the data set unless i had very good reason to. I also don't think you'll get predictive data out of it, but it may depend on how you implement it. Overall if you were to do this it would seem to be more of a hobby thing to see if it even even come close, like "oh neat i got a neural network to within X amount of accuracy". Again, i must stress, don't use this in any sort of production situations or anywhere where it could possibly hurt or kill someone!
We are looking at updating (rewriting) our system which stores information about when people can reserve rooms etc. during the day. Right now we store the start and time and the date the room is available in one table, and in another we store the individual appointment times.
On the surface it seemed like a logical idea to store the information this way, but as time progressed and the system came under heavy load, we began to realize that this data structure appears to be inefficient. (It becomes an intensive operation to search all rooms for available times and calculate when the rooms are available. If the room is available for a given time, is the time that it is available long enough to accommodate the requested time).
We have gone around in circles about how to make the system more efficient, and we feel there has to be a better way to approach this. Does anyone have suggestions about how to go about this, or have any places where to look about how to build something like this?
I found this book to be inspiring and a must-read for any kind of database involving time management/constraints:
Developing Time-Oriented Database Applications in SQL
(Added by editor: the book is available online, via the Richard Snodgrass's home page. It is a good book.)
#Radu094 has pointed you to a good source of information - but it will be tough going processing that.
At a horribly pragmatic level, have you considered recording appointments and available information in a single table, rather than in two tables? For each day, slice the time up into 'never available' (before the office opens, after the office closes - if such a thing happens), 'available - can be allocated', and 'not available'. These (two or) three classes of bookings would be recorded in contiguous intervals (with start and end time for each interval in a single record).
For each room and each date, it is necessary to create a set of 'not in use' bookings (depending on whether you go with 'never available', the set might be one 'available' record or it might include the early shift and late shift 'never available' records too).
Then you have to work out what questions you are asking. For example:
Can I book Room X on Day Y between T1 and T2?
Is there any room available on Day Y between T1 and T2?
At what times on Day Y is Room X still available?
At what times on Day Y is a room with audio-visual capabilities and capacity for 12 people available?
Who has Room X booked during the morning of Day Y?
This is only a small subset of the possibilities. But with some care and attention to detail, the queries become manageable. Validating the constraints in the DBMS will be harder. That is, ensuring that if the time [T1..T2) is booked, then no-one else books [T1+00:01..T2-00:01) or any other overlapping period. See Allen's Interval Algebra at Wikipedia and other places (including this one at uci.edu).