In AWS, can I get disk (volume) images sent to me on DVD?
This would be in preparation to save costs by eliminating the instances and volumes. It took a year to get everything created, the right instances and structure hierarchy. And experimentation to select the right combinations of instances and volume types.
My company wants to save the monthly costs (for a year or so) but also wants the option and ability to restart everything when they want.
Is it possible to be able to save the whole set-up, remove all the instances and volumes (to eliminate costs), and re-institute it back when needed?
Depending on your instance operating system (RedHat, Windows, etc.) you might not be able to get images of the EC2 instance in any form other than AMIs or snaphots stored at Amazon. The reason is Amazon's license does not permit this.
I have not heard of AWS offering a service to back up your data to DVDs. If your goal is to save money, think of the cost per hour that AWS would charge you to manually create DVDs. The DVD write process is slow ....
Instead approach the situation of either creating Amazon Machine Images (AMIs) of your instances or creating snapshots of your instances. You would need to pay the cost of data storage which is about $.05 per GB. 100 GB of data would cost $60.00 per year. 100 GB on DVDs that AWS generated would be significantly higher if offered.
You only can export an image if you import in the first place (https://docs.aws.amazon.com/vm-import/latest/userguide/vmexport.html)
I recommend just creating snapshots, it won't bring the cost to zero but it's cheaper than having stopped instances with unused volumes. If volumes are General Purpose SSD (gp2), snapshots will be half of the price ($0.05 per GB-month)
I definitely don't recommend storing data on DVDs, take advantage of the cloud and the resilience it provides. I'm sure these DVDs won't work when you try to use them after a year if you can even find them, that will cost more than staying on the cloud. :)
Related
I have a use case where we have a very large computation job, which can be broken up into many small units of work fairly efficiently. There could be effectively lets say 1,000 hours of computational work for an m4.large instance. Lets say I wanted the result back within the next 10 minutes, that would mean I would need 6,000 instances to get the job done in time.
So far I have setup AWS batch, I haven't used any more than the 20 m4.large instances your account comes with. I know I can up the amount of instances requested by AWS but I still don't really know much about what the behaviour is if you suddenly try and provision thousands of on-demand instances or if AWS limits how many instances you can use.
So my question is am I able to launch thousands of m4.large instances on-demand? And if so what are sort of times would I be looking at for all instances to get to the Running state.
I have done this many times with ~100 instances but never in the thousands of instances.
STEP 1: Open a support ticket with AWS. You will need to get your account approved, credit checked, etc. My customers are very big companies, so for them the credit and approval process is easy. If you are a little guy, I don't know.
STEP 2: Think thru your VPC design and how you will address that many instances. If is one thing to have 5 instances going thru a NAT Gateway, but a hundred systems will bring Internet connectivity to its knees.
STEP 3: Think thru the networking bandwidth required. Do you need placement groups or very high speed Intranet or Internet connectivity?
STEP 4: Be prepared that you cannot launch all instances with a specific instance type (capacity not available error). Have a selection of instances that you can fall back on.
STEP 5: Create your own software, I use Python, to launch the instances, perform updates, install software, etc. You can then poll the instances using the Boto3 EC2 API to determine when all the instances are running. The length of time for 1,000 instances won't be much different than 1 instance.
Now for the real world. If your job takes 1,000 hours, launching 1,000 instances will not reduce it to 1 hour unless you have a really scalable software design with minimum inter-machine communications required. Once you go beyond 10 systems, networking bandwidth and communications overhead becomes an issue. Even though AWS's resources are huge, launching 1,000 EC2 instances at one time by one customer is not a common launch case.
I would also NOT launch 1,000 instances to get processing down to 10 minutes. It can take 10 minutes for your instances to come online, get updated, synchronize, etc. This means that you will be spending 50% of your budget on waiting time. For really large jobs today we prefer to use Hadoop / Spark where scaling to hundreds of machines is realistic.
You can contact AWS Customer Service to increase your EC2 limits (use the link shown in the Limits section of the EC2 management console). They will verify your use-case.
You might also consider using Spot Pricing to lower your costs. Spot instances take longer to provision.
Sample use-case: Gigaom | Cycle Computing once again showcases Amazon’s high-performance computing potential
There are also services like Spotinst that can help you provision servers at the lowest possible cost.
As I understand it, RDS Provisioned IOPS is quite expensive compared to standard I/O rate.
In Tokyo region, P-IOPS rate is 0.15$/GB, 0.12$/IOP for standard deployment. (Double the price for Multi-AZ deployment...)
For P-IOPS, the minimum required storage is 100GB, IOP is 1000.
Therefore, starting cost for P-IOPS is 135$ excluding instance pricing.
For my case, using P-IOPS costs about 100X more than using standard I/O rate.
This may be a very subjective question, but please give some opinion.
In the most optimized database for RDS P-IOPS, would the performance be worth the price?
or
The AWS site gives some insights on how P-IOPS can benefit the performance. Is there any actual benchmark?
SELF ANSWER
In addition to the answer that zeroSkillz wrote, I did some more research. However, please note that I am not an expert on reading database benchmarks. Also, the benchmark and the answer was based on EBS.
According to an article written by "Rodrigo Campos", the performance does actually improve significantly.
From 1000 IOPS to 2000 IOPS, the read/write(including random read/write) performance doubles. From what zeroSkillz said, the standard EBS block provices about 100 IOPS. Imagine the improvement on performance when 100 IOPS goes up to 1000 IOPS(which is the minimum IOPS for P-IOPS deployment).
Conclusion
According to the benchmark, the performance/price seems reasonable. For performance critical situations, I guess some people or companies should choose P-IOPS even when they are charged 100X more.
However, if I were a financial consultant in a small or medium business, I would just scale-up(as in CPU, memory) on my RDS instances gradually until the performance/price matches P-IOPS.
Ok. This is a bad question because it doesn't mention the size of the allocated storage or any other details of the setup. We use RDS and it has its pluses and minuses. First- you can't use an ephemeral storage device with RDS. You cant even access the storage device directly when using the RDS service.
That being said - the storage medium for RDS is presumed to be based on a variant of EBS from amazon. Performance for standard IOPS depends on the size of the volume and there are many sources stating that above 100GB storage they start to "stripe" EBS volumes. This provides better average case data access both on read and write.
We run currently about 300GB of storage allocation and can get 2k write IOP and 1k IOP about 85% of the time over a several hour time period. We use datadog to log this so we can actually see. We've seen bursts of up to 4k write IOPs, but nothing sustained like that.
The main symptom we see from an application side is lock contention if the IOPS for writing is not enough. The number and frequency you get of these in your application logs will give you symptoms for exhausting the IOPS of standard RDS. You can also use a service like datadog to monitor the IOPS.
The problem with provisioned IOPS is they assume steady state volumes of writes / reads in order to be cost effective. This is almost never a realistic use case and is the reason Amazon started cloud services to fix. The only assurance you get with P-IOPS is that you'll get a max throughput capability reserved. If don't use it, you pay for it still.
If you're ok with running replicas, we recommend running a read-only replica as a NON-RDS instance, and putting it on a regular EC2 instance. You can get better read-IOPS at a much cheaper price by managing the replica yourself. We even setup replicas outside AWS using stunnel and put SSD drives as the primary block device and we get ridiculous read speeds for our reporting systems - literally 100 times faster than we get from RDS.
I hope this helps give some real world details. In short, in my opinion - unless you must ensure a certain level of throughput capability (or your application will fail) on a constant basis (or at any given point) there are better alternatives to provisioned-IOPS including read-write splitting with read-replicas memcache etc.
So, I just got off of a call with an Amazon System Engineer, and he had some interesting insights related to this question. (ie. this is 2nd hand knowledge.)
standard EBS blocks can handle bursty traffic well, but eventually it will taper off to about 100 iops. There were several alternatives that this engineer suggested.
some customers use multiple small EBS blocks and stripe them. This will improve IOPS, and be the most cost effective. You don't need to worry about mirroring because EBS is mirrored behind the scenes.
some customers use the ephemeral storage on the EC2 instance. (or RDS instance) and have multiple slaves to "ensure" durabilty. The ephemeral storage is local storage and much faster than EBS. You can even use SSD provisioned EC2 instances.
some customers will configure the master to use provisioned IOPS, or SSD ephemeral storage, then use standard EBS storage for the slave(s). Expected performance is good, but failover performance is degraded (but still available)
anyway, If you decide to use any of these strategies, I would recheck with amazon to make sure I haven't forgotten any important steps. As I said before, this is 2nd hand knowledge.
I'm new to everything that is 'the cloud.'
I will be developing a website/platform that will have around 15,000,000 estimated monthly visitors after the first year of production.
I'm assuming that the site will have 5 page views per visitor, and 100kb of data transfer per page.
I've contacted several cloud hosting companies, but they tell me that I need to have 'hardware requirements.'
Since I'm rather clueless about IT stuff, I'd like to know:
What are the factors that need to be analyzed in order to determine
How many servers are required
VPUs / server required
RAM / server required
Total storage / server required
Big thanks in advance!
I don't agree with the other answer as it's nearly total guesswork, as will anything you can generate yourself.
The only surefire way to know is to get some hardware, stick your application on it and run some load testing to see if you can get to the point you want to traffic wise, and with a certain amount of free overhead on the servers. Only then will you know what you need. No-one else can answer this question as every application is different. This is your application, only you can test it.
Data given wont help much in determining what numbers you want. But based on my experience I'll try to help you in analysis.
15,000,000 visits a month means 700K visits a day (assuming approx 30-35% visits are by repeat visitors).
700Kx5=3.5million page views a day.
Assuming 14 hours of active period, typical for single timezeone sites. Its 70reqs/sec.
With this big userbase few thing you surely need is a high performance DB server, with one slave.
Config of these DB server
Memory so that whole active data + indexes fits in memory (No swapping/thrashing should happen). This you need to calculate based on
what you will be storing for user and for how long.
Use some reliable storage like RAID10 (higher read/write bandwith).
Take enough storage, see that its elastic enough. (like AWS EBS).
Make frontend app server lightweight and horizontally scalable. Put them behind a loadbalancer (use software loadbalancer like nginx or HAproxy). You should be able to put as many as you go to your goal.
For loadbalacer and frontend take 4CPU, 4-8GB RAM servers.
How much each frontend can take need to be tested using a load testing method and realistic test data.
Reduce load on database/persistent using a inmemory/+persistent caches like memcached/membase/redis etc. Take a servers with 8GB and add more as you feel need.
I have not discussed about DB partitioning. Do that only when you feel the need of it. Do not over invest at start.
With 15M users a month, this setup should be enough, but again it all depends on you 1. memory footprint, 2. amount of active data
I tried to answer as much as possible. Comments on points you disagree or wanna discuss more.
Just a simple question, say I on a research team with 10 members and we want to deploy a project that will require on-demand leasing of 200 EC2 instances. Can we virtually bypass the 20 instances limit by creating 10 AWS accounts (one each) and then lease on demand up to 200 instances?
You can do that - you can have as many accounts, with their appropriate payment details, as you like. You then have the standard 20-instance limit associated with each account.
Bear in mind though, that these accounts are all separate and have their own sets of access keys. Any technology you deploy to span instances across these accounts will need to be aware of that, since your 'pool' manager will need the appropriate key for the account it is attempting to launch an instance in. You may also encounter complexities with EBS sharing, Security Group access (if, for example, you have a Domain Controller in one account, VMs in the other accounts will not be able to see it), and Load Balancing and VPC are likely to be difficult at best.
Amazon are usually fairly quick to respond to limit increase requests - after all, the more instances you're using, the more money they're making. They obviously want to ensure they have capacity for your requirement, which is why they manually 'vet' requests, but provided you have a reasonable business case they're fairly accomodating.
http://aws.amazon.com/ec2/faqs/#How_many_instances_can_I_run_in_Amazon_EC2
You are limited to running 20 On-Demand or Reserved Instances, and
running 100 Spot Instances per region. Cluster GPU Quadruple Extra
Large instances and High I/O Quadruple Extra Large instances are
limited to running 2 On-Demand instances per region [...].
Increase your limit with http://aws.amazon.com/contact-us/ec2-request
Bid the 'on-demand price' for a bunch of spot instances. You will never end up paying full price, but will rarely have your spot instances terminated. See
http://aws.amazon.com/ec2/spot-instances/ and deciding on your spot bidding strategy
What gives best performance for running PostgreSQL on EC2? EBS in RAID? PGData on /mnt?
Do you have any preferences or experiences? Main "plus" for running PostgreSQL on EBS is switching from one to another instance. Can this be the reason to be slower than using the /mnt partition?
PS: I'm running PostgreSQL 8.4 with datas/size about 50G, Amazon EC2 xlarge(64) instance.
Here there is some linked info. The main take-away is this post from Bryan Murphy:
Been running a very busy 170+ gb OLTP postgres database on Amazon for
1.5 years now. I can't say I'm "happy" but I've made it work and still prefer it to running downtown to a colo at 3am when something
goes wrong.
There are two main things to be wary of:
1) Physical I/O is not very good, thus how that first system used a RAID0.
Let's be clear here, physical I/O is at times terrible. :)
If you have a larger database, the EBS volumes are going to become a
real bottleneck. Our primary database needs 8 EBS volumes in a RAID
drive and we use slony to offload requests to two slave machines and
it still can't really keep up.
There's no way we could run this database on a single EBS volume.
I also recommend you use RAID10, not RAID0. EBS volumes fail. More
frequently, single volumes will experience very long periods of poor
performance. The more drives you have in your raid, the more you'll
smooth things out. However, there have been occasions where we've had
to swap out a poor performing volume for a new one and rebuild the
RAID to get things back up to speed. You can't do that with a RAID0
array.
2) Reliability of EBS is terrible by database standards; I commented on this
a bit already at
http://archives.postgresql.org/pgsql-general/2009-06/msg00762.php The end
result is that you must be careful about how you back your data up, with a
continuous streaming backup via WAL shipping being the recommended approach.
I wouldn't deploy into this environment in a situation where losing a
minute or two of transactions in the case of a EC2/EBS failure would be
unacceptable, because that's something that's a bit more likely to hapen
here than on most database hardware.
Agreed. We have three WAL-shipped spares. One streams our WAL files
to a single EBS volume which we use for worst case scenario snapshot
backups. The other two are exact replicas of our primary database
(one in the west coast data center, and the other in an east coast
data center) which we have for failover.
If we ever have to worst-case-scenario restore from one of our EBS
snapshots, we're down for six hours because we'll have to stream the
data from our EBS snapshot back over to an EBS raid array. 170gb at
20mb/sec (if you're lucky) takes a LONG time. It takes 30 to 60
minutes for one of those snapshots to become "usable" once we create a
drive from it, and then we still have to bring up the database and
wait an agonizingly long time for hot data to stream back into memory.
We had to fail over to one of our spares twice in the last 1.5 years.
Not fun. Both times were due to instance failure.
It's possible to run a larger database on EC2, but it takes a lot of
work, careful planning and a thick skin.
Bryan