I have an AWS account with some small instances on it.
Is there a way to see the amount of data which has been transfetered (in/out) per instance?
Thanks
Dmi
If the instances are still running, you can do this using the free basic monitoring that Amazon provides via CloudWatch.
For Amazon EC2 instances, Amazon CloudWatch Basic Monitoring collects
and reports metrics for CPU utilization, data transfer, and disk usage
activity from each Amazon EC2 instance at a five-minute frequency.
To access the monitor data for an EC2 instance, select it in the AWS Management Console and click the "Monitoring" tab in the lower panel. You can click any of the charts (e.g. "Max Network In") to get a larger version, with some drop-downs to control the time window.
Note that the data is only kept for two weeks.
Related
I have a java process on AWS. It is not dockerised for business reasons. But if the process goes down. It should spawn a new EC2 instance. Is this possible with Cloud watch?
What you are looking for is AWS Autoscaling.
AWS Auto Scaling is a service that automatically monitors and adjusts
compute resources to maintain performance for applications hosted in
the Amazon Web Services (AWS) public cloud.
Essentially what it does is, it integrates with AWS Cloud Watch to set alarms for specific situations.
For example, if the instance gets hit with x number of requests within a time period of y mins. Or for example, the health check of an instance fails (your situation).
When you are launching your EC2 instance, explore and check out Autoscaling groups and Launch Configuration. Check out this link below
https://docs.aws.amazon.com/autoscaling/ec2/userguide/GettingStartedTutorial.html
Now you mentioned that there is a process which is running in your EC2 instance.
Launch Configurations give you a way to provide a bash script which is executed when your instance starts.
This bash script should contain the logic to pull your business logic from wherever you are hosting it, and then run that process.
Let me know if you have any confusions.
I found the answer in the below link. AWS Cloud watch will help us monitor applications and reboot EC2 instance if application doesn't respond.
https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-action-reboot-ec2-instance/
I would like to use a EC2 instance for accounting purpose as the instance is required to be accessed by multiple party. Since EC2 charges are per hour basis. I am planning to turn on the instance only when some one needs to use then turn if off. The instance is not required to be live 24x7.
Start the instance with a mouse click. (maybe some script or panel or url.
Remote Desktop Connect- My accountant finishes the work.
User shutdown the instance or auto shutdown after 30 mins of inactivity.
How do we go about it ?
Thanks,
Dwija
You can Start an Amazon EC2 instance that was previously Stopped by sending a StartInstance command. This can be done from many different programming languages, or via the AWS Command-Line Interface (CLI) (available for Windows, Mac and Linux).
Turning off the instance when it is not in use is harder because you need to identify when it is appropriate to turn it off. You could have a program on the instance itself (much like a screensaver) that detects when the system is 'idle' and issues a Shutdown command on the virtual machine (just like Choosing Shutdown from the Start menu). This will Stop the Amazon EC2 instance.
An alternative, depending on your use-case, would be to use Amazon Workspaces, which is a virtual desktop -- a computer that is running in the cloud that you can connect with from Windows, Mac, iOS or Android devices. It is charged either on a monthly subscription or on a combination of subscription + hourly charge. It automatically turns off after the user has disconnected for a period of time. An Amazon Workspace is typically used by one person as their 'personal' computer, but it could be shared amongst other users.
If I stop an instance, I understand that I am not charged. However, I've installed several items on my machine in the last few hours (Mysql uploaded a db, Apache, PHP, etc...). I assume that when I stop the instance I retain these software installations, correct? Are these things stored in EBS? If so, where is the pricing listed for EBS, as I guess I will be charged every hour of every day for the whole year for my EBS storage given that it can never "stop"...
Thanks in advance.
From http://aws.amazon.com/ec2/pricing/, the current rates are:
$0.10 per GB-month of provisioned storage
$0.10 per 1 million I/O requests
Your data will persist if the root device is EBS backed, which is default now. You can check in the EC2 control panel.
So stopping the instance (if 8GB EBS) will cost about $0.80 per month for storage if it is never running.
If your instance is EBS backed all data is safe. You will be charged for EBS storage only. Amazon EBS storage charged on per-month basis. Currently it's $0.10 per GB-month.
I am deploying a rails application on EC2 instances, where I want to configure Auto Scaling to be used with an Elastic Load Balancer. On a particular threshold I want to spawn a new instance. While configuring trigger for auto scaling group, we have options for CPU Utilization, Network or disk IO; but being a rails application, I will face resource crunch on memory rather than CPU or IO.
Has anyone configured EC2 Auto Scaling for a rails application successfully? What is the preferred way of using AWS with rails?
FYI: I am using passenger as application server.
Thanks for your time.
I have not done it with Rails, but with java in Tomcat.We used tomcat valves/scripts to detect the memory usage and post it in Amazon cloudWatch as Custom cloud watch metrics. You can create an scale out trigger based on cloudwatch alarm monitoring this metric.
Some sections of the above technique can even be overlapped for rails.
Actually, I think you should try to tune your Passenger configuration based on the instance type you are using (here is an article about this: http://blog.scoutapp.com/articles/2009/12/08/production-rails-tuning-with-passenger-passengermaxprocesses). This should ensure that at full load you are using all the RAM available but you are not spawning more passenger instances than RAM available.
This, in my experience leads to saturating all the resources (CPU & RAM) and you could trigger an autoscale policy based on CPU usage. You also should tweak the instance type in order to achieve the best performance (I've used cc1.xlarge instance with a fair amount of success).
If you're set on autoscaling based on RAM, you should be able to create a metric on CloudWatch that is monitoring RAM usage and autoscale using that metric. Creating a metric is just publishing the metric data at regular intervals using the CloudWatch API (http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/CloudWatch/Metric.html). You could create a rails background task that runs every minute and publishes the metric data.
i think you can use the new signaling feature, monitor the memory metric and start the instance using api:
http://aws.typepad.com/aws/2010/12/amazon-cloudwatch-alarms.html
We're running a lightweight web app on a single EC2 server instance, which is fine for our needs, but we're wondering about monitoring and restarting it if it goes down.
We have a separate non-Amazon server we'd like to use to monitor the EC2 and start a fresh instance if necessary and shut down the old one. All our user data is on Elastic Storage, so we're not too worried about losing anything.
I was wondering if anyone has any experience of using EC2 in this way, and in particular of automating the process of starting the new instance? We have no problem creating something from scratch, but it seems like it should be a solved problem, so I was wondering if anyone has any tips, links, scripts, tutorials, etc to share.
Thanks.
You should have a look at puppet and its support for AWS. I would also look at the RightScale AWS library as well as this post about starting a server with the RightScale scripts. You may also find this article on web serving with EC2 useful. I have done something similar to this but without the external monitoring, the node monitored itself and shut down when it was no longer needed then a new one would start up later when there was more work to do.
Couple of points:
You MUST MUST MUST back up your Amazon EBS volume.
They claim "better" reliability, but not 100%, and it's SEVERAL orders of magnitude off of S3's "12 9's" of durability. S3 durability >> EBS durability. That's a fact. EBS supports a "snapshots" feature which backs up your storage efficiently and incrementally to S3. Also, with EBS snapshots, you only pay for the compressed deltas, which is typically far far less than the allocated volume size. In another life, I've sent lost-volume emails to smaller customers like you who "thought" that EBS was "durable" and trusted it with the only copy of a mission-critical database... it's heartbreaking.
Your Q: automating start-up of a new instance
The design path you mention is relatively untraveled; here's why... Lots of companies run redundant "hot-spare" instances where the second instance is booted and running. This allows rapid failover (seconds) in the event of "failure" (could be hardware or software). The issue with a "cold-spare" is that it's harder to keep the machine up to date and ready to pick up where the old box left off. More important, it's tricky to VALIDATE that the spare is capable of successfully recovering your production service. Hardware is more reliable than untested software systems. TEST TEST TEST. If you haven't tested your fail-over, it doesn't work.
The simple automation of starting a new EBS instance is easy, bordering on trivial. It's just a one-line bash script calling the EC2 command-line tools. What's tricky is everything on top of that. Such a solution pretty much implies a fully 100% automated deployment process. And this is all specific to your application. Can your app pull down all the data it needs to run (maybe it's stored in S3?). Can you kill you instance today and boot a new instance with 0.000 manual setup/install steps?
Or, you may be talking about a scenario I'll call "re-instancing an EBS volume":
EC2 box dies (root volume is EBS)
Force detach EBS volume
Boot new EC2 instance with the EBS volume
... That mostly works. The gotchas:
Doesn't protect against EBS failures, either total volume loss or an availability loss
Recovery time is O(minutes) assuming everything works just right
Your services need to be configured to restart automatically. It does no good to bring the box back if Nginx isn't running.
Your DNS routes or other services or whatever need to be ok with the IP-address changing. This can be worked around with ElasticIP.
How are your host SSH keys handled? Same name, new host key can break SSH-based automation when it gets the strong-warning for host-key-changed.
I don't have proof of this (other than seeing it happen once), but I believe that EC2/EBS _already_does_this_ automatically for boot-from-EBS instances
Again, the hard part here is on your plate. Can you stop your production service today and bring it up RELIABLY on a new instance? If so, the EC2 part of the story is really really easy.
As a side point:
All our user data is on Elastic Storage, so we're not too worried about losing anything.
I'd strongly suggest to regularly snapshot your EBS (Elastic Block Storage) to S3 if you are not doing that already.
You can use an autoscale group with a min/max/desired quantity of 1. Place the instance behind an ELB and have the autoscale group be triggered by the ELB healthy node count. This allows you to have built in monitoring by cloudwatch and the ELB health check. Anytime there is an issue the instance be replaced by the autoscale service.
If you have not checked 'Protect against accidental termination' you might want to do so.
Even if you have disabled 'Detailed Monitoring' for your instance you should still see the 'StatusCheckFailed' metric for your instance over which you can configure an alarm (In the CloudWatch dashboard)
Your application (hosted in a different server) should receive the alarm and start the instance using the AWS API (or CLI)
Since you have protected against accidental termination you would never need to spawn a new instance.