I sampled 1000 people in 2018 and then 1000 people in 2019.
For brand A, 5% of participants like the brand in 2018, and 8% of participants like the brand in 2019.
For brand B, 25% of participants like the brand in 2018, and 29% of participants like the brand in 2019.
So people who like the brand grow 3% for Brand A, and 4% for Brand B, and I want to know if this 1% difference is significant or not, what's the right method to do the test?
Thank you!
Related
I'm playing a little with spot instances, and for example, in Databricks, I can ask for a spot instance with a minimum of % savings over On-Demand instances.
My question is, if I set 90% off the On-Demand instance and the current price is 50%, I will get the cheaper instance or is it like bidding and I get the spot for 90% price?
I have some use cases when the availability of one instance is not very important so will be good to get those at any discount.
Summing up, if I set a minimum of 90% I will always get the cheaper spot available?
Thanks!
As per the this article from databricks:
Databricks defaults to a bid price that is equal to the current on-demand price. So if the on-demand price for your instance type is $0.665 cents per hour, then the default bid price is also set to $0.665 cents per hour.
Recall that with Spot instances, you don’t necessarily pay your actual bid price - you pay the Spot market price for your instance type. So, even though you may have bid $0.665 cents, in reality you may only be charged $0.10 for the hour.
So its safe to assume that you will be charged whats the current spot market price but not more then what you set.
I have to do performance testing of an ecommerce application where i got the details needed like Avg TPH and peak TPH . also Avg User and Peak User.
for e.g., an average of 1000 orders/hour, the peak of 3000 orders/hour during the holiday season, expected to grow to 6000 orders/hour next holiday season.
I was afraid which value to be considered for current users and TPH for performing load test for an hour.
also what load will be preferable foe stress testing and scalability testing.
It would be a great helpful not only for the test point of view but also will help me in understanding the conceptually which would help me in a great deal down the lane.
This is a high business risk endeavor. Get it wrong and your ledger doesn't go from red to black on the day after thanksgiving, plus you have a high probability of winding up with a bad public relations event on Twitter. Add to that greater than 40% of people who hit a website failure will not return.
That being said, do your skills match the risk to the business. If not, the best thing to do is to advise your management to acquire a higher skilled team. Then you should shadow them in all of their actions.
I think it helps to have some numbers here. There are roughly 35 days in this year's holiday shopping season. This translates to 840 hours.
#$25 avg sale, you are looking at revenue of $21 million
#$50 avg sale, ...42 Million
#100 avg sale, ...84 Million
Numbers based upon the average of 1000 sales per hour over 840 hours.
Every hour of downtime at peak costs you
#$25 avg sale, ...$75K
#$50 avg sale, ...$150K
#$100 avg sale, ...$300K
Numbers based upon 3000 orders per hour at peak. If you have downtime then greater than 40% of people will not return based upon latest studies. And you have the Twitter affect where people complain loudly and draw off potential site visitors.
I would advise you to bring in a team. Act fast, the really good engineers are quickly being snapped up for Holiday work. These are not numbers to take lightly nor is it work to press someone into that hasn't done it before.
If you are seriously in need and your marketing department knows exactly how much increased conversion they get from a faster website, then I can find someone for you. They will do the work upfront at no charge, but they will charge a 12 month residual based upon the decrease in response time and the increased conversion that results
Normally Performance Testing technique is not limited to only one scenario, you need to run different performance test types to assess various aspects of your application.
Load Testing - which goal is to check how does your application behave under anticipated load, in your case it would be simulating 1000 orders per hour.
Stress Testing - putting the application under test under maximum anticipated load (in your case 3000 TPH). Another approach is gradually increasing the load until response time starts exceeding acceptable thresholds or errors start occurring (whatever comes the first) or up to 6000 TPH if you don't plan to scale up. This way you will be able to identify the bottleneck and determine what will be the component which fails which could be in
lack of hardware power
problems with database
inefficient algorithms used in your application
You can also consider executing a Soak Test - putting your application under prolonged load, this way you will be able to catch the majority of memory leaks
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 5 years ago.
Improve this question
I am trying to get my head around the Evidence Based Scheduling (EBS) approach used in FogBugz and I have read Evidence Based Scheduling several times.
What I do understand is the general idea, why Monte-Carlo is used, and so on ...
And I can also calculate the extrapolation of an estimation by using the factor's of the past stories. So far, so good.
Question 1
The question I have is: How do I calculate the probability distribution for more than one story? I.e., I want to know when five stories will be finished.
May I just add up the 10% values, the 20% values, ..., and finally the 100% values?
To give an example:
I know that story 1 is estimated as 4 hours, and its probability distribution tells me that 0% is 3 hours, 25% is 4 hours, 50% is 4 hours, 75% is 5 hours, and 100% is 9 hours.
I know that story 2 is estimated as 6 hours, and its probability distribution tells me that 0% is 4 hours, 25% is 6 hours, 50% is 6 hours, 75% is 7 hours, and 100% is 13 hours.
If I now want to know the probability distribution of story 1 and 2, may I just add them, so I get:
0%: 7 hours
25%: 10 hours
50%: 10 hours
75%: 12 hours
100%: 22 hours
Is that all I need to do? Or is it something more complicated?
Question 2
My other question is how to calculate the end time for multiple tasks when there is more than user involved, but I do not know in advance which user will work on what story. As long as I know that assignment, it's quite easy: Calculcate the sum of stories for each user, and then take the latest one as an overall-time (if one finishes after 3 weeks, the other after 5 weeks, the total project will take 5 weeks).
But what if I don't know in advance, and not every user is able to work on every story? E.g., I have put competencies onto stories, such as front-end, back-end, ... and I have assigned competencies to my users, so there may be developers for front-end, for back-end, ... and so on.
Of course there may be stories which require multiple competencies, which in return requires work from multiple users. But they will be working on different things and require different times for finishing their tasks. And this again depends on the probability distribution: If one has a run, he might finish earlier than if he didn't have. This may influence on what he will work next, whom he may assist, and so on ...
Any idea of how I could calculate this?
1.
You may not add up the values at corresponding levels in the probability distributions. That would baselessly assume perfect correlation between the task completion times. Here is something that you may do instead.
In the worst case, the time to complete two tasks is the sum of the times to complete each task. So far, so good. Worst-case estimation in software development is probably just fine. I don't think that totaling up the times will generally cause a problem.
Now we need to consider whether the two tasks' times are governed by probability distributions that are independent of one other. That is, when we know anything about how soon one task is completed, does that tell us something about how soon the other task is completed? If we don't know, can we make a suitably safe assumption?
Of course it depends on the cost of incorrect estimation, but it may be safe enough to assume that the distributions are indeed independent. That way, at least the completion of one task generally doesn't give us false hope about the other. So the answer is, if one task is analyzed into M outcomes each with its own probability, and the other task is analyzed into N outcomes each with its own probability, we can form the M*N outcomes and assign to the (i,j) outcome the product of probability (density) of the i-th outcome of the first task with the probability (density) of the j-th outcome of the second task.
I'm going to modify your example because, sorry, I don't understand it. Let's say that the first task has this distribution instead, where X is a uniformly distributed continuous random variable between 0% and 100%:
3 hours, if X <= 20% (with probability density 20%);
4 hours, if 20% < X <= 60% (with probability density 40%);
5 hours, if 60% < X <= 80% (with probability density 20%);
9 hours, if 80% < X (with probability density 20%).
The second task has this distribution, where Y is a uniformly distributed continuous random variable between 0% and 100%, independent of X:
4 hours, if Y <= 20% (with probability density 20%);
6 hours, if 20% < Y <= 60% (with probability density 40%);
7 hours, if 60% < Y <= 80% (with probability density 20%);
13 hours, if 80% < Y (with probability density 20%).
Now we calculate as follows:
4#20% 6# 40% 7#20% 13#20%
------ -------- ------- --------
3#20% | 3+4# 4% 3+6# 8% 3+7# 4% 3+13# 4%
4#40% | 4+4# 8% 4+6# 16% 4+7# 8% 4+13# 8%
5#20% | 5+4# 4% 5+6# 8% 5+7# 4% 5+13# 4%
9#20% | 9+4# 4% 9+6# 8% 9+7# 4% 9+13# 4%
So here's the probability distribution and density for the sum of the two tasks' times, where Z is a uniformly distributed continuous random variable from 0% to 100%:
7 hours, if Z <= 4% (with probability density 4%);
8 hours, if 4% < Z <= 12% (with probability density 8%);
9 hours, if 12% < Z <= 24% (with probability density 12%);
10 hours, if 24% < Z <= 44% (with probability density 20%);
11 hours, if 44% < Z <= 60% (with probability density 16%);
12 hours, if 60% < Z <= 64% (with probability density 4%);
13 hours, if 64% < Z <= 68% (with probability density 4%);
15 hours, if 68% < Z <= 76% (with probability density 8%);
16 hours, if 76% < Z <= 84% (with probability density 8%);
17 hours, if 84% < Z <= 92% (with probability density 8%);
18 hours, if 92% < Z <= 96% (with probability density 4%);
22 hours, if 96% < Z (with probability density 4%).
All of this may be tedious, but it's logical and not hard to automate.
2.
You are correct, there is a fanning out of scenarios. Roughly, it starts with the initial certainty that before the world existed, nobody had yet done anything! After that, well, after you have automation for question 1, you could employ various strategies in your analysis. Maybe your imagination is as good as mine for this purpose. Anyway, here's what I can suggest.
You could explore what-if scenarios interactively.
You could attempt to compute and total up everything that could possibly happen. As we have seen, this kind of analysis is possible for small cases. As we can imagine, it will become intractible in large cases, such as presumably building a flight navigation system.
You could analyze the most likely scenario and perhaps a limited degree of variation around that.
Very likely, you will be interested in controlling your risks. So you could consider analyzing one or more of the following, according to your needs and convenience, all of them being a bit different from the rest: the chance of an unacceptable outcome, or the chance that unacceptable degree of uncertainty exists, or an estimate of how much uncertainty exists, or an estimate of the expected outcome (that is, the average outcome if one were to face the same situation endlessly repeated).
Without Googling the problem, I would guess that the "unknown developer capabilities" probably pushes the problem into the NP-hard optimization problems bin. A couple of algorithms to look at are Simulated Annealing and Genetic Algorithms (Simulated Annealing was used in Wintek's electronic CAD autoplacement program (I was on the development team)).
They talk about the different utilization instances, but I can't find a single place that actually states what the difference is between a light utilization Large instance or a heavy utilization Large instance other than price. Can someone please tell me what the difference is?
The instances are the same. It's just a pricing difference so you can save money if you know you will be using the instance a lot, by paying an upfront cost.
You pay more up front for a heavier utilization instance, but you save more in the long run assuming you have it running all the time, because the hourly rate is cheaper.
So it's just a matter of how much you will be using the instance (having it running) that determines which type is the best value for you. If it will be on all the time for a year or 3 years, then a heavy utilization is definitely the cheapest option.
on demand, light, medium, and heavy offer different up-front vs hourly costs, with progressively higher up-front costs, and lower per-hour costs..
according to http://aws.amazon.com/ec2/reserved-instances/
Light Utilization RIs – The break-even point for a Light Utilization Linux RI (vs. On-Demand Instances) is 28% for a 1-year term or 11% of a 3-year term. If you expect to use your instance more than that, an RI will save you money.
Medium Utilization RIs – The break-even point for a Medium Utilization Linux RI (vs. Light Utilization Reserved Instances) is 41% for a 1-year term or 19% of a 3-year term.
Heavy Utilization RIs – The break-even point for a Heavy Utilization Linux RIs (vs. Medium Utilization Reserved Instances) is 56% for a 1-year term or 35% of a 3-year term.
Here's a link to Amazon's documentation:
http://aws.amazon.com/ec2/reserved-instances/#2
It doesn't explain/quantify what amount of usage constitutes Light, Medium, or Large but it gives a ballpark amount.
To summarize (excerpted from the above documentation):
Light Utilization RIs are ideal for periodic workloads that only run a couple of hours a day, a few days per week, or very sporadically
Medium Utilization RIs are best suited for workloads that run most of the time, but have some variability in usage (like web server traffic where demand may increase or decrease throughout the year)
Heavy Utilization RIs offer the most absolute savings of any Reserved Instance type. They’re most appropriate for steady-state workloads where you’re willing to commit to always running these instances in exchange for our lowest hourly usage fee
If you're like me and wanted to see what we're getting for the prices we're paying for Light, Medium, and Heavy, here it is.
I think the answers are already great here. Just a tip that may help you to select the right reservation type is http://www.cloudorado.com/ . Go to advanced mode and then select how long you plan to run service (subscription duration) and how long the server will run per day. The service will choose the best reservation option - you can see it in details.
The math here gets a little complicated. But, at the end of the day, it comes down to how many hours you'll be running instances for the reservation type/size/region that you have. It's gotten even more complicated now that you can trade in reservations for different AZ's and sizes.
There's a great blog series on choosing Reserved Instance types/quantities on the Cloudability blog.
http://cldy.co/choosing-ri-types
There's also a webinar recording that covers all the math:
http://cldy.co/ri-webinar-recording
Full disclosure: I work for Cloudability.
I am new to the Oracle 10g Resource Manager and am looking for guidance on how to put together a plan to meet my goals. I will test the plan, but I'm hoping for some guidance so I don't have to try hundreds of different plan configurations. I have the following goals:
Don't let non-sys sessions significantly slow down sys sessions.
Don't let any OLTP users sessions significantly slow down any other OLTP users sessions.
Don't let batch processing slow down OLTP.
Given these goals my first thought is to create the following consumer groups/plan directives:
Consumer Group Level 1 Level 2 Level 3
SYS 100% 0% 0%
OLTP1 0% 10% 0%
OLTP2 0% 10% 0%
OLTP3 0% 10% 0%
OLTP4 0% 10% 0%
OLTP5 0% 10% 0%
OLTP6 0% 10% 0%
OLTP7 0% 10% 0%
OLTP8 0% 10% 0%
OLTP9 0% 10% 0%
OLTP10 0% 10% 0%
BATCH 0% 0% 100%
Using this method each OLTP user could be put in a different OLTP group (assuming 10 users). The documentation isn't very clear on this, but it sounds like if an OLTP user in group OLTP1 needs more than it's 10% share that it will get it as long as every other OLTP group is getting 10% if it needs it. Is my understanding accurate? Will this work or is there a better way?
I would simplify this a little, make one group for OLTP, if they have the same requirements. Only make a new group when that new group has different requirements than the others in terms of priority. Also make sure that when an OLTP user has started a long running heavy duty process, that this session is switched to the batch group, or not started at all.
Resource manager only kicks in when cpu consumption is at 100%. From that point on it will start dividing resources to make sure that each group gets what it should get based on your directives.
Other things to think of are max parallel degree, session pool and (from 11g and up) undo usage and io limits.
best regards,
Ronald
http://ronr.blogspot.com