When autoscaling my EC2 instances for application, what is the best way to keep every instances in sync?
For example, there are custom settings and application files like below...
Apache httpd.conf
php.ini
PHP source for my application
To get my autoscaling working, all of these must be configured same in each EC2 instances, and I want to know the best practice to sync these elements.
You could use a private AMI which contains scripts that install software or checkout the code from SVN, etc.. The second possibility to use a deployment framework like chef or puppet.
The way this works with Amazon EC2 is that you can pass user-data to each instance -- generally a script of some sort to run commands, e.g. for bootstrapping. As far as I can see CreateLaunchConfiguration allows you to define that as well.
If running this yourself is too much of an obstacle, I'd recommend a service like:
scalarium
rightscale
scalr (also opensource)
They all offer some form of scaling.
HTH
Related
My organization's website is a Django app running on front end webservers + a few background processing servers in AWS.
We're currently using Ansible for both :
system configuration (from a bare OS image)
frequent manually-triggered code deployments.
The same Ansible playbook is able to provision either a local Vagrant dev VM, or a production EC2 instance from scratch.
We now want to implement autoscaling in EC2, and that requires some changes towards a "treat servers as cattle, not pets" philosophy.
The first prerequisite was to move from a statically managed Ansible inventory to a dynamic, EC2 API-based one, done.
The next big question is how to deploy in this new world where throwaway instances come up & down in the middle of the night. The options I can think of are :
Bake a new fully-deployed AMI for each deploy, create a new AS Launch config and update the AS group with that. Sounds very, very cumbersome, but also very reliable because of the clean slate approach, and will ensure that any system changes the code requires will be here. Also, no additional steps needed on instance bootup, so up & running more quickly.
Use a base AMI that doesn't change very often, automatically get the latest app code from git upon bootup, start webserver. Once it's up just do manual deploys as needed, like before. But what if the new code depends on a change in the system config (new package, permissions, etc) ? Looks like you have to start taking care of dependencies between code versions and system/AMI versions, whereas the "just do a full ansible run" approach was more integrated and more reliable. Is it more than just a potential headache in practice ?
Use Docker ? I have a strong hunch it can be useful, but I'm not sure yet how it would fit our picture. We're a relatively self-contained Django front-end app with just RabbitMQ + memcache as services, which we're never going to run on the same host anyway. So what benefits are there in building a Docker image using Ansible that contains system packages + latest code, rather than having Ansible just do it directly on an EC2 instance ?
How do you do it ? Any insights / best practices ?
Thanks !
This question is very opinion based. But just to give you my take, I would just go with prebaking the AMIs with Ansible and then use CloudFormation to deploy your stacks with Autoscaling, Monitoring and your pre-baked AMIs. The advantage of this is that if you have most of the application stack pre-baked into the AMI autoscaling UP will happen faster.
Docker is another approach but in my opinion it adds an extra layer in your application that you may not need if you are already using EC2. Docker can be really useful if you say want to containerize in a single server. Maybe you have some extra capacity in a server and Docker will allow you to run that extra application on the same server without interfering with existing ones.
Having said that some people find Docker useful not in the sort of way to optimize the resources in a single server but rather in a sort of way that it allows you to pre-bake your applications in containers. So when you do deploy a new version or new code all you have to do is copy/replicate these docker containers across your servers, then stop the old container versions and start the new container versions.
My two cents.
A hybrid solution may give you the desired result. Store the head docker image in S3, prebake the AMI with a simple fetch and run script on start (or pass it into a stock AMI with user-data). Version control by moving the head image to your latest stable version, you could probably also implement test stacks of new versions by making the fetch script smart enough to identify which docker version to fetch based on instance tags which are configurable at instance launch.
You can also use AWS CodeDeploy with AutoScaling and your build server. We use CodeDeploy plugin for Jenkins.
This setup allows you to:
perform your build in Jenkins
upload to S3 bucket
deploy to all the EC2s one by one which are part of the assigned AWS Auto-Scaling group.
All that with a push of a button!
Here is the AWS tutorial: Deploy an Application to an Auto Scaling Group Using AWS CodeDeploy
I am very new to the Amazon AWS services. I was wondering if there is a way to run an instance of EC2 (say, Amazon Linux AMI) and then connect two environments to this instance.
Particularly, I'd like to run a PHP and a Tomcat environment on a single EC2 instance.
The problem is, every time I create a new environment in Elastic Beanstalk, it seems to create a new EC2 instance as well. Am I missing something here?
I'd appreciate any hint on this.
AWS Elastic Beanstalk is designed for deploying your running apps in a way that is designed for scalability from the ground-up. Because of this, Elastic Beanstalk will launch one or more EC2 instances, connect them to an Elastic Load Balancer instance, configure CloudWatch monitoring and Auto Scaling triggers.
Also, because of its fundamental design for scalability, Elastic Beanstalk is designed around a one-app-per-environment model (whereby "environment", I mean one of these EC2 + ELB + CloudWatch + AutoScaling clusters).
Since running two separate web servers with two separate apps (PHP & Java) is not a fundamentally scalable design, it's not a use-case that Elastic Beanstalk is optimized for.
You are free to spin-up a standalone EC2 instance and install whatever you'd like on it, but you're right — git aws.push support has not been made available for standalone EC2 instances. If the git support is important to you, you'll need to weigh the pros and cons of each approach.
I would also like to be able to do this, basically from a cost perspective for demos etc.
For example, a single instance with one PHP app and one Java app. Or, a single instance with two Java apps.
However, from what I have read so far in the Elastic Beanstalk developer guide, I have not found anything explicitly stating that multiple applications per environment is supported (or even, multiple environments per EC2 instance - if that even makes sense).
It makes me wonder if this is a feature that is often requested and planned for the future, or alternatively if the single-app-per-environment model is 'by design' for some reason.
I am currently experimenting with Amazon EC2 and use standard ec2 console. The web app is ok but I want a better solution. I want to be able to ssh to the instances, monitor them, possibly attach with a debugger etc. Are there any better alternatives to the tool?
You should be able to login to any EC2 instance via SSH using key files and work with it like if it were an ordinary server. To do it you have to create a key pair, download public key to your local machine, and ensure you've selected that key-pair while launching new instance. You are free to install any software you like on the instance, so the way how you would monitor you instance is completely up to you (if you decide not to use AWS console).
Apart from the web console there are also Amazon EC2 API tools (a bunch of ec2 scripts to be run from Linux console) and the Query API. The later is considered to be the most flexible way to manage your cloud infrastructure. There are binding for EC2 in many scripting languages including Python (boto), Perl (Net::Amazon::EC2), Ruby (amazon-ec2 gem), node.js (aws2js).
Otherwise there's no better solution just because EC2 is IaaS service and it is meant to be equally good for almost any task. For your particular needs you'll have to develop or organize your own environment which will suite your unique needs.
Edit:
Since today it is possible to log in to running EC2 Linux instances from AWS web console:
Our third announcement today is about a new feature in the AWS console that makes it even easier for you to use Amazon EC2 Linux instances. Customers have been asking us to enable the ability to log into their instances directly from the AWS console. Starting today, you can log in to your Linux instances from the EC2 console without the need to install additional software clients. Please see the Amazon EC2 Getting Started Guide for details on how to use this new functionality.
I usually use Munin as monitoring software, but this (as others software I presume) needs an IP to make the ICMP or whatever pings to collect data.
In Amazon EC2 instances are created on the fly, with IP's you don't know.
How can they be monitored ?
I was thinking about using amazon console commands to read the IP's of the instances up, and change the monit configuration file on the fly also , but it can be too complicated ... or not?
Any other solution / suggestion ?
Thank you
I use revealcloud to monitor my amazon instances. You can install it once and create an ami from that systen, or bootstrap the install command if that's your method. Since the install is just one command, it's easy enough to put into the rc.local (or similar). You can then see all the instances in the dashboard or topiew as soon as they boot up.
Our instances are bootstrapped using chef recipes, so it's easier for me to provide IPs/hosts as they (= all members of my cluster) get entered into /etc/hosts on start-up. Generally, it doesn't hurt to use elastic IPs for a master server and allow all connections (in /etc/munin/munin.conf by default).
I'd solve the security 'question' on the security groups level. E.g. allow only instances with a certain security group to connect to the munin-node process (on port 4949). The question which remains is.
E.g., using ec2-authorize you can achieve
ec2-authorize mygroup -o monitorgroup -u <AWS-USER-ID>
This means that all instances with group monitorgroup can access resources on instances with mygroup.
Let me know if this helps!
If your Munin master and nodes are all hosted on EC2 than it's better to use internal hosts like domU-00-00-00-00-00-00.compute-1.internal. because this way you don't have to deal with IP addresses and security groups.
You also have to set this in /etc/munin/munin-node.conf:
allow ^.*$
You can read more about it in Monitoring AWS Ubuntu Instances using Munin
But if your Munin master is not on EC2 your best bet is to attach Elastic IP to your EC2 instance.
Reading about and using the Amazon Web Services, I'm not really able to grasp how to use it correctly. Sorry about the long question:
I have a EC2 instance which mostly does the work of a web server (apache for file sharing and Tomcat with Play Framework for the web app). As it's a web server, the instance is running 24/7.
It just came to my attention that the data on the EC2 instance is non persistent. This means I lose my database and files if it's stopped. But I guess it also means my server settings and installed applications are lost as they are just files in the same way as the other data.
This means that I will either have to rewrite the whole app to use amazon CloudDB or write some code which stores the db on S3 and make my own AMI with the correct applications installed and configured. Or can this be quick-fixed by using EBS somehow?
My question is 1. is my understanding of aws is correct? and 2. is it's worth it? It could be a possibility to just set up a regular dedicated server where everything is persistent, as you would expect. Would love to have the scaleability of aws though..
If you use an EBS volume with your EC2 instance, you can mount/dismount them to have persistent storage. You can also use Amazon RDS to handle your database too which is handy (but can be slightly on the pricier side.)
So a way to think of it is:
Your EC2 instance: Get the OS set up exactly like you'd like it along with your web application - basically, get your static stuff all in place.
EBS volume: That can be mounted and can be used for things like user uploads.
RDS instance: This is a dedicated database server with no hassles. It's nice - I use a MySQL RDS and it automatically makes two daily backups, and is scalable like EC2 instances.
Amazon Web Service is a better approach at hosting your applications Jon. You have a basic understand of AWS but you need to know that you can also launch an instance that is persistent. Just launch an instance of a persistence AMI. Also you can install you database,webs server on the instance like a regular server. There is probably just minimal differences from running an Ec2 instance and a dedicated server. If you have any other questions you can contact me.