I'm using an ECS cluster that is running on EC2 instances, I have several instances of my app running as tasks in the cluster. I want to add a cache layer for my app, this layer will write the data in the disk. Furthermore, I also want to know how much memory an AWS ECS task will give to my container? and what happens to the files that I had in my container after a deployment?.
I already look in google for answers, but I only found information for tasks that are running in a Fargate cluster.
Related
I am confused about how Service AutoScaling automately works. Will it create EC2 instance automately?
I create it and add it to a Cluster's service, but it does no create EC2 for placing my required number of tasks. Is any thing wrong with my settings? I check the [Events] and see "service s2 was unable to place a task because no container instance met all of its requirements. ", but shouldn't it create a EC2 if no instance met? Please give me some advice, thanks in advance.
but shouldn't it create a EC2 if no instance met
Not really. There are two types of scaling policies: scaling policies on an ECS service and scaling policies on the ECS cluster. Instances are added based on cluster scaling policies, and that's what you should set up in addition to your service scaling policy.
AWS has a couple of detailed tutorials on scaling ECS clusters:
https://aws.amazon.com/blogs/compute/automatic-scaling-with-amazon-ecs/
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch_alarm_autoscaling.html
AWS Elastic Container Services has two methods to deploy containers over aws environment
Where you no need to worry about orchestration of containers (task in aws)
Fargate (Available in few regions like N.Virginia)
Using EC2 in ECS
I guess you are using 2nd option to deploy application over ECS where you can provide details of scaling tasks/containers not ec2 instances.
For Auto-scaling of ec2 instances you should look into ASG of AWS.
As far as AWS ECS is concerned you need some building blocks which are as follows-
Cluster
Task definition (Memory, Network and Storage configs of tasks)
Service contains EC2 instance configuration
Auto scaling policies if you want to auto-scale tasks
I have created my own EC2 instance in AWS. That AMI is AWS ECS optimized AMI for launching ecs service from my EC2 instance. I previously discussed the same thing. And tried with that approach. The link is below,
Microservice Deployment Using AWS ECS Service
I created my cluster and configured that cluster name when I am creating optimized AMI by following code snippet in advanced userdata section,
#!/bin/bash
echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config
I followed the documentation of cluster creation from following link,
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create_cluster.htmlecs
But, no result - when creating cluster and ECS task definitions it creates and launches into one EC2. And again creating another EC2 by specifying above code. So total 2 Ec2. I already created my own ECS optimized.
I am finding for launching ECS service from my own AMI (that I created). Actually I need to launch my ECS service from my Ec2 (I had created my machine Amazon optimized AMI).
The reason behind this requirement is I don't want to launch my services in machine that owned by others. I need to launch from my machine. And also I need to host my angular application in the same my machine. So I need control of my machine. How can I do this?
Sounds like you just need to create a Launch Configuration. With this you can specify the User Data settings that should be applied when a host is setup.
After you create your Launch Configuration, create a new Auto Scaling Group based off of it (there's a drop-down to select the launch configuration you want to use).
From here, any new instances launched under that ASG will apply the settings you've configured in the associated Launch Configuration.
How can stateful containers be run inside Mesos?
According to the Mesos documentation sandbox can be used to store state:
With the introduction of persistent volumes, executors and tasks
should never create files outside of the sandbox.
At the same time Sandbox files are scheduled for garbage collection when:
An executor is removed or terminated.
A framework is removed.
An executor is recovered unsuccessfully during agent recovery.
Is this the only way? Or can docker containers be used to maintain state (in a similar manner to a VM)?
So for example, can a container be created and run across 2 nodes? Can such a container contain state and not be disposed of after the task is completed?
The key statement in that quote from the Mesos documentation is
With the introduction of persistent volumes...
You're correct that sandboxes can be garbage collected. However, Mesos provides a primitive called persistent volumes which allows you to create volumes that will persist across task failures and agent restarts and will not be garbage collected.
Additionally, Mesos also now provides support for network storage via the Docker volume isolator. This allows you to mount network volumes using Docker volume drivers, which enables the use of a wide variety of storage back-ends.
Docker containers can store persistent state, but they must do so in either a Mesos persistent volume or a network-attached volume via the Docker volume isolator. These volumes live outside the Docker container and are mounted into the container, so they persist after the container has died.
Mesos tasks cannot be run across multiple nodes. Note that it would be possible for multiple tasks on different nodes to access the same network-attached volume via the Docker volume isolator, provided the back-end storage provider supports concurrent access.
So, I'm trying to understand CPU and VM allocation with kubernetes, docker and AWS ecs.
Does this seem right?
Locally, running "docker compose" with a few services:
each container gets added to the single Docker Machine VM. You can allocate CPU shares from this single VM.
AWS, running ECS, generated from a docker compose:
each container (all of them) gets added to a single ec2 VM. You can allocate CPU shares from that single VM. The fact that you deploy to a cluster of 5 ec2 instances makes no difference unless you manually "add instances" to your app. Your 5 containers will be sharing 1 ec2.
AWS, running kubernetes, using replication controllers and service yamls:
each get container gets distributed amongst ALL of your ec2 instances in your kubernetes cluster?????
If i spin up a cluster of 5 ec2 instances, and then deploy 5 replication controllers / services, will they be actually distributed across ec2's? this seems like a major difference from ECS and local development. Just trying to get the right facts.
Here are the answers to your different questions:
1> Yes you are right,you have a single VM and any container you run will get cpu shares from this single VM. You also have the option of spawning a swarm cluster and try out. Docker compose support swarm for containers connected via a overlay network spread over multiple vms.
2> Yes your containers defined in a single task will end up in the same ec2 instance. When you spin up more than one instances of the task, the tasks get spread over the instances part of the cluster. Non of tasks should have resource requirement which is greater than the max resource available on one of your ec2 instances.
3> Kubernetes is more evolved than ECS in many aspects, but in case of container distribution it works similar ecs. Kubernetes pod is equivalent to a ecs task. Which is one or a group of container colocated on a single VM. In kubernetes also you cannot have a pod need resources more the max available on one of your underneath compute resources.
In all the three scenarios, you are bound by the max capacity available on underneath resource when deploying a large container or a pod.
You should not equate the docker platform to VM creation and management platform.
All these docker platforms expect you to define tasks which fit into the VMs and require you to scale horizontally with more task count when needed. Kubernetes comes with service discovery, which allows seamless routing of requests to the deployed containers using DNS lookups. You will have build your own service discovery with swarm and ecs. CONSUL, EUREKA etc are tools which you can use for the same.
I am quite new to Amazon services, and started reading about EMR. I am more or less familiar with OpenStack. I just want some one to tell me in short what plays the role of Compute, Controller and Cinder of storage in Amazon cloud.
For example Cinder is storage for OpenStack and likewise S3 is the storage in Amazon cloud.
What are the the other two - compute and controller in Amazon cloud?
Also, can some 1 please put up in simple words the relation between EMR and EC2 or are they entirely different ?
Even in EMR we use EC2 instances, so why are people comparing hadoop on EC2 vs Map Reduce like in the following link
Hadoop on EC2 vs Elastic Map Reduce
Thanks a ton in advance :)
Openstack is an open source software that can be setup in your own cloud so that you can have your managed services like Amazon.
Amazon is it's own independent service with its own proprietary implementation and they basically sell the service.
So Openstack has several components that has a somehow 1-1 mapping with AWS services.
Controller -> Amazon Console
Cinder -> EBS
Storage -> S3
Compute -> EC2
EMR (Elastic Map Reduce) is just another service from Amazon that allows you to run hadoop jobs. EMR basically runs on top of EC2 so in essence when you create an EMR cluster it's using EC2 as its underlying service.
You can also run Hadoop independently from EMR on EC2 instances, the downside is that you have to manage all the Hadoop installation, configuration yourself (Cloudera manager is pretty helpful for this). The advantage is that it allows you to tweak as much as you want from the Hadoop stack.
Hope this helps.