I maintain alot of amazon instances of different types (medium, micro, small) etc. However, Some instances I want to reserve for a few years. But,
From what I've read everywhere :
Reserved instances are a billing feature, not a technical feature.
So if I reserve a micro instance and later I convert that instance from micro to large, will the reserved instance slot be taken up by the next micro instance I have in my ec2 instances list (provided they are in the same AZ) ?
Thanks.
Yes, it is basically as you say. When you are billed Amazon first checks for reserved instances and only if you are out of those, normal rates apply.
See here for reference:
"Easy to Use: Reserved Instances are easy to use and require no change to how you use EC2. When computing your bill, our system will automatically apply Reserved Instance rates first to minimize your costs. An instance hour will only be charged at the On-Demand rate when your total quantity of instances running that hour exceeds the number of applicable Reserved Instances you own."
So technically there is no difference between an reserved instance and a non-reserved one.
It will be taken by any instance that matches the reservation. A reservation is not tied to a specific instance.
Yes.
I like to call it the trickle down. Where the reserved instance will trickle down to the next unreserved instance that matches the same instance size.
Note: One nice feature is the ability to reorganize your instances to different sizes within the same instance type. For example, you have 5 m1.xlarge instances, you can split them into 40 m1.small instaces and vice versa. Unfortunately, you can't go between m1, t1, c1, c3, etc.
Related
I have been checking all EC2 instances and.. from all types I'm debating between choosing -reserved instance or dedicated instance-
Spot and on demand doesn't meet the requirement I think.
I don't have more information about the type of app to be run.
It sounds like you are concerned with the availability (or up time) of different EC2 instances. The difference between reserved, dedicated, spot and on-demand has more to do with cost than availability. AFAIK AWS does not guarantee different levels of availability for different EC2 instance types or cost structures.
Dedicated instances are ones that won't run on the same hardware as other instances, but dedicated instances can run on the same hardware as other non-dedicated instances in the same account. Dedicated instances shouldn't have any affect on availability.
The other options, (reserved, spot and on-demand) are just different cost structures. They don't affect performance or availability.
AWS advertises 99.99% uptime. This is from their SLA:
AWS will use commercially reasonable efforts to make the Included Services each available for each AWS region with a Monthly Uptime Percentage of at least 99.99%, in each case during any monthly billing cycle (the “Service Commitment”). In the event any of the Included Services do not meet the Service Commitment, you will be eligible to receive a Service Credit as described below.
So, any instance will be appropriate for important long running apps, you just need to select the instance type that is large enough and pick the right cost structure. You probably don't need dedicated instances, unless you have determined that steel time is going to impact your app's performance.
I dont think its Instance Types you need to worry about - they are just different sizes of machines that have different ratios of CPU/RAM/IO. High-Availability can only be achieved by designing it into both the application and infrastructure. Reservation Types (spot/on-demand/dedicated) are more about cost optimisation based on how long and much capacity you need.
Spot instances are transient - you can reserve a spot for upto 6 hours then it will be terminated. Probably not a good fit, but it depends on how long is "long running"? You get a significant discount for using spot instances if you can live with the limitations.
On-Demand and Reserved are basically the same service, but on-demand is pay-as-you-go vs reserved instance is a fixed length contract and upfront payments. Reserved instances give you the best value for money over the long term, but on-demand gives you flexibility to change instance size or turn off the instance to save money.
Going shared vs Dedicated is generally a compliance question as it gets a lot more expensive to reserve your own hardware to run just your own instances.
I have a use case where we have a very large computation job, which can be broken up into many small units of work fairly efficiently. There could be effectively lets say 1,000 hours of computational work for an m4.large instance. Lets say I wanted the result back within the next 10 minutes, that would mean I would need 6,000 instances to get the job done in time.
So far I have setup AWS batch, I haven't used any more than the 20 m4.large instances your account comes with. I know I can up the amount of instances requested by AWS but I still don't really know much about what the behaviour is if you suddenly try and provision thousands of on-demand instances or if AWS limits how many instances you can use.
So my question is am I able to launch thousands of m4.large instances on-demand? And if so what are sort of times would I be looking at for all instances to get to the Running state.
I have done this many times with ~100 instances but never in the thousands of instances.
STEP 1: Open a support ticket with AWS. You will need to get your account approved, credit checked, etc. My customers are very big companies, so for them the credit and approval process is easy. If you are a little guy, I don't know.
STEP 2: Think thru your VPC design and how you will address that many instances. If is one thing to have 5 instances going thru a NAT Gateway, but a hundred systems will bring Internet connectivity to its knees.
STEP 3: Think thru the networking bandwidth required. Do you need placement groups or very high speed Intranet or Internet connectivity?
STEP 4: Be prepared that you cannot launch all instances with a specific instance type (capacity not available error). Have a selection of instances that you can fall back on.
STEP 5: Create your own software, I use Python, to launch the instances, perform updates, install software, etc. You can then poll the instances using the Boto3 EC2 API to determine when all the instances are running. The length of time for 1,000 instances won't be much different than 1 instance.
Now for the real world. If your job takes 1,000 hours, launching 1,000 instances will not reduce it to 1 hour unless you have a really scalable software design with minimum inter-machine communications required. Once you go beyond 10 systems, networking bandwidth and communications overhead becomes an issue. Even though AWS's resources are huge, launching 1,000 EC2 instances at one time by one customer is not a common launch case.
I would also NOT launch 1,000 instances to get processing down to 10 minutes. It can take 10 minutes for your instances to come online, get updated, synchronize, etc. This means that you will be spending 50% of your budget on waiting time. For really large jobs today we prefer to use Hadoop / Spark where scaling to hundreds of machines is realistic.
You can contact AWS Customer Service to increase your EC2 limits (use the link shown in the Limits section of the EC2 management console). They will verify your use-case.
You might also consider using Spot Pricing to lower your costs. Spot instances take longer to provision.
Sample use-case: Gigaom | Cycle Computing once again showcases Amazon’s high-performance computing potential
There are also services like Spotinst that can help you provision servers at the lowest possible cost.
I'm considering purchasing an Amazon AWS EC2 reserved instance. Does any one know of any performance benefits for reserved vs. on-demand, spot instances? Or is this simply a pricing difference? Do reserved instances get spun up on VMs with less congestion and that sort of thing is what I'm thinking about.
Thanks,
Michael
Despite its name, "reserved instance" is a billing construct, not an instance type. When you purchase a reserved instance, no instance is started. It simply affects the price you pay for a standard on-demand instance that matches the availability zone, instance type, operating system of the reserved instance purchase.
Purchasing a reserved instance can reduce the hourly price you pay for an on-demand instance that has already been running for months. If you terminate that on-demand instance, the reserved instance pricing switches to any other on-demand instance that is running and matches the specs.
Spot instances are the same as on-demand instances except in when they are started/terminated and how much they cost.
Spot instances are basically when amazon has unused resources they can sell them at a slightly lower price to hopefully get those used up. The issue with spot instances is that they can be shut off at any time so you're taking a risk.
Reserved instances are computing power you can purchase upfront at a discounted rate. You're basically saying that you're going to use this type of system for 1 or 3 years or if you're not going to use it you want to ensure that you can use it at any time.
On-demand is the default for most people, I think. They're there. They're available when you need them and when you don't need them you stop paying for them.
If you purchase the same types of machine configs for each type of instance then there is no performance difference, only price.
The Amazon page on reserved instances doesn't mention any other benefits apart from the cost.
Adding to what #BuildStarted said, I have noticed longer start times with spot instances. I'm guessing that's because there is some additional time needed to compute the current spot price and find capacity.
Once an instance is up and running, though, you shouldn't see any difference between the 3 pricing methods for the same instance type.
Just a simple question, say I on a research team with 10 members and we want to deploy a project that will require on-demand leasing of 200 EC2 instances. Can we virtually bypass the 20 instances limit by creating 10 AWS accounts (one each) and then lease on demand up to 200 instances?
You can do that - you can have as many accounts, with their appropriate payment details, as you like. You then have the standard 20-instance limit associated with each account.
Bear in mind though, that these accounts are all separate and have their own sets of access keys. Any technology you deploy to span instances across these accounts will need to be aware of that, since your 'pool' manager will need the appropriate key for the account it is attempting to launch an instance in. You may also encounter complexities with EBS sharing, Security Group access (if, for example, you have a Domain Controller in one account, VMs in the other accounts will not be able to see it), and Load Balancing and VPC are likely to be difficult at best.
Amazon are usually fairly quick to respond to limit increase requests - after all, the more instances you're using, the more money they're making. They obviously want to ensure they have capacity for your requirement, which is why they manually 'vet' requests, but provided you have a reasonable business case they're fairly accomodating.
http://aws.amazon.com/ec2/faqs/#How_many_instances_can_I_run_in_Amazon_EC2
You are limited to running 20 On-Demand or Reserved Instances, and
running 100 Spot Instances per region. Cluster GPU Quadruple Extra
Large instances and High I/O Quadruple Extra Large instances are
limited to running 2 On-Demand instances per region [...].
Increase your limit with http://aws.amazon.com/contact-us/ec2-request
Bid the 'on-demand price' for a bunch of spot instances. You will never end up paying full price, but will rarely have your spot instances terminated. See
http://aws.amazon.com/ec2/spot-instances/ and deciding on your spot bidding strategy
I am looking at the pricing of various cloud computing platforms, particularly Amazon's EC2, and a lot of the quotes are based on a unit called an Instance-Hour.
I am trying to get a handle on the exact definition of an instance-hour to better compare the costs of continuing to host a web-application versus putting it out on the cloud.
(1) Does it correspond to any of the Windows performance counters in such a way that I could benchmark our current implmentation and use it in their pricing calculators?
(2) How does a multi-processor instance figure into the instance-hour calculation?
An instance hour is simply a regular hour where the instance was available to you, wether you used it or not. Amazon has priced their different types of instances differently, so you pay for the type of resource you are getting, not how much you use it.
So... 1. No, it's just a regular hour. 2. It doesn't, it's already factored into the price you pay for the instance pr hour.
Note also that instance hours are billed rounded up (for Amazon EC2). So starting up an instance and immediately shutting it down again incurs the cost of 1 instance hour.
if you plan to start/stop the AWS EMR cluster rapidly within a single hour, and you want to avoid being billed for a full hour each time you do so, then
start the cluster with the --alive argument from cli, which means to leave it running.
then rapidly add steps to the same cluster, instead:
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/AddingStepstoaJobFlow.html
Don't forget to stop the cluster when you're done! :)