How can I terminate default EC2 instance. I can delete the default instance but automatically it is getting created after some time
Is it part of a auto scaling group? Auto scaling groups has a parameter called MinSize. This is the minimum size of the group.
Type: Integer
Required: Yes
If the number of instance fails below MinSize it'll automatically create one.
Also make sure there are no automated jobs that are creating instances using AWS CLI.
Related
I have an Android and IOS app that uploads images (about 15,000 per minute) to a AWS S3 bucket, everything is all right, but i need to process those images in a web app that is used from 2 to 50 different users called 'Monitores' , when this kind of user logins and begin to process the images the app scan the S3 bucket for the filenames, something like:
$recibidos = Storage::disk('s3recibidos');
$total_archivos = $recibidos->allfiles();
this generates an array with the files are stored in the time the route is invoked, if i use this with one user for process there is no problem, because the process is one time only, but what if i have 2 or more users trigger this process? the process retrieves no the exact list but i think many of the un processed files will be duplicated.
The process of the filenames is to store in a database and to move to a subdirectory.
For example:
I have 1000 files in the AWS S3 bucket and user1 invoke the process so the array will have 1000 filenames to process, right now the time to process those files is about 3 min, so before the process finish 1000 new files was added to the AWS S3 bucket this files are not in the user1 array, then user2 logins and begins to process, so right now the AWS S3 has new files and old files, then when get the new array gets some old filenames (the ones are not process), in fact when user2 process the files some of this was not available, because the user1 process was made the job.
I need help in this two things:
1.- How to deal with the process.
2.- How can i use wildcards, because one of the final process changes the filename of the files in S3, so the filename list that i need to process has its exepecific format.
Thanks for any advice
I'm a little confused about your process, but let's assume:
You have a large number of incoming images
You need to perform some operation on each of those instances
There are two recommended approaches to do this:
Option 1: Serverless
Configure the Amazon S3 bucket to trigger an AWS Lambda function whenever a new object is created in the bucket
Create an AWS Lambda function as a worker -- it receives information about each file, then processes the file
AWS Lambda will automatically scale to run multiple Lambda functions in parallel. The default is up to 1000 concurrent Lambda functions, but this can be increased upon request.
Option 2: Traditional
Create an Amazon SQS queue to store details of images to process
Configure the Amazon S3 bucket to send an event to the SQS queue whenever a new object is created in the bucket
Use Amazon EC2 instance(s) to run multiple workers
Each worker reads the file information from the queue, processes the image, then deletes the message from the queue. It then repeats, pulling the next message from the queue.
Scale the number of EC2 instances and/or workers as necessary
Both of these approaches have workers operating on one image files at a time, so you do not have the problem of maintaining lists while images being continually added. They are also highly scalable with no code changes.
To reduce costs I would like to stop and start an container instance in a cluster in-between tasks. The task run every now and again so doesn't seem efficient keeping an EC2 running in-between.
What is the best way to allow this?
I have looked into lambda functions triggered by a cloudwatch scheduler and also thought about autoscaling.
Amazon doesn't make this incredibly straight-forward (though, they're trying to with Fargate). The best solution for now (if you're not in a region where Fargate is an option) is to try to keep your desired task count in line with your desired instance count on an autoscaling group.
The way we have it setup is through Lambda, triggering based on Autoscaling events (pretty easy to setup). The least trivial part about this is the Lambda script, though it's not incredibly difficult. Add tags to your ASG that help identify what cluster / service it's associated with. When a scaling event triggers your script, just have your script describe the ASG that triggered it, look for the cluster / service that's in the tags, and updated the desired count of that service:
asgDetail = paginator_describe_asg.paginate(
AutoScalingGroupNames=[
asgName,
]
)
# set service desired count equal to ASG desired capacity
newDesiredCount = iter(asgDetail).next()['AutoScalingGroups'][0]['DesiredCapacity']
response = client_ecs.update_service(
cluster = <ecs cluster>,
desiredCount = newDesiredCount,
service = <ecs service>
)
The reason you shouldn't rely on CloudWatch for this is because it doesn't do a great job at granular scaling. What I mean is, the CPU that CloudWatch monitors on your ASG is the overall group average (I think). So the scenario we ran into was as follows:
CloudWatch detects hosts are at 90%, desired is 70%
CloudWatch launches 4 hosts
Service detects tasks are at 85%, desired is 70%
Service launches new task
Service detects tasks are at 80%, desired is 70%
Service launches new task
Service detects tasks are at 75%, desired is 70%
Service launches new task
Service detects tasks are at 70%, no action
While this is a trivial example, it's easy to see how the number of instances get out of sync from the number of tasks actually running (i.e., you may end up with a host sitting idle because ECS doesn't detect that it needs more capacity).
Could we just scale up 3 hosts? Sure, but ECS might still only place 2 tasks (depending on how the usage is per task). Could we scale one host at a time? Sure, but then it's pretty difficult to account for bursts.
All that to say, the best solution I can recommend for now is to have a Lambda script help keep your ASG instance count == your ECS service desired task count.
I have decided to create a lambda function that starts the instance and on container instance start a task is ran. Then I have a cloud watch event watching for the task changing status to STOPPED which triggers another lambda that stops the instance.
I want to stop EC2 instances after office hours to save costs. How can I do the same with ECS instances? Even if I stop all tasks/services, the instance is still there? Do I stop the EC2 instance directly?
From EC2 Management Console
Click Auto Scaling Groups from the left menu.
Select the group from the list.
Click edit on the details tab.
Set desired property to '0'.
After clicking save it is all done.
The Auto Scaling Group is smart enough to shut down all instances.
You can use the "Scheduled Actions" feature of Auto Scaling Groups.
Starts similar to Kerem Baydoğan's answer
From EC2 Management Console:
1 Click Auto Scaling Groups from the left menu.
2 Select the group from the list.
3 Select "Scheduled Actions" from the bar that appeared in the lower middle of the screen.
4 Click on create scheduled action
5 Fill the fields as you see fit and notice that under recurrence there is also a cron option for extra flexibility.
If you have the cluster set to a minimum number of nodes with an asg. If you turn off the nodes the asg will start another node to bring it up to three minimum number of nodes. You must set the asg to zero nodes.. Then turn off the current nodes.
Yes, just stop the EC2 instance directly. When you start the instance again during office hours, the ECS agent will make the services start according to their desired value.
We are doing the same thing and it works for us.
I have an EC2 instance that I want to scale based on the number of messages in a SQS queue. If there are many messages (for 5 minutes) I want to pop up a new EC2, for consuming faster the messages. Then if the messages are few (for 5 minutes), I want to pop down the oldest EC2. This way, if the service that consumes the messages stops for some reason, I will terminate the old EC2, and the service will run.
I have created an AutoScalling for this. I have set the TerminationPolicy to OldestInstance, but it works as I expect only if I set just one zone (eg: eu-west-1a): it creates a new instance and terminates the oldest each time. But if I have 3 regions (eu-west-1a, eu-west-1b, eu-west-1c), it just launches and terminates the instances not in the OldestInstance manner. Or, at least, not as I expect: delete the oldest every time. Is there something linked to different zones? On this pace I have not found anything about it, except for the default policy.
And even if the case linking to multiple zones from default policy is applied, I can have maximum only 2 instances that turn at the same time. And they are always launched in a new zone.
This is probably the key paragraph:
When you customize the termination policy, Auto Scaling first assesses the Availability Zones for any imbalance. If an Availability Zone has more instances than the other Availability Zones that are used by the group, then Auto Scaling applies your specified termination policy on the instances from the imbalanced Availability Zone. If the Availability Zones used by the group are balanced, then Auto Scaling selects an Availability Zone at random and applies the termination policy that you specified.
I interpret this to mean that if you have instances in multiple zones, and those zones are already balanced, then AWS will select a zone at random AND THEN pick the oldest instance, within the randomly selected zone - it won't pick the oldest instances across AZ's, it picks and random AZ and then the oldest instance is terminated within that AZ.
I am trying to find out how many MASTER, CORE, TASK instances are optimal to my jobs. I couldn't find any tutorial that explains how do I figure it out.
How do I know if I need more than 1 core instance? What are the "symptoms" I would see in EMR's console in the metrics that would hint I need more than one core? So far when I tried the same job with 1*core+7*task instances it ran pretty much like on 8*core, but it doesn't make much sense to me. Or is it possible that my job is so much CPU bound that the IO is such minor? (I have a map-only job that parses apache log files into csv file)
Is there such a thing to have more than 1 master instance? If yes, when is it needed? I wonder, because my master node pretty much is just waiting for the other nodes to do the job (0%CPU) for 95% of the time.
Can the master and the core node be identical? I can have a master only cluster, when the 1 and only node does everything. It looks like it would be logical to be able to have a cluster with 1 node that is the master and the core , and the rest are task nodes, but it seems to be impossible to set it up that way with EMR. Why is that?
The master instance acts as a manager and coordinates everything that goes in the whole cluster. As such, it has to exist in every job flow you run but just one instance is all you need. Unless you are deploying a single-node cluster (in which case the master instance is the only node running), it does not do any heavy lifting as far as actual MapReducing is concerned, so the instance does not have to be a powerful machine.
The number of core instances that you need really depends on the job and how fast you want to process it, so there is no single correct answer. A good thing is that you can resize the core/task instance group, so if you think your job is running slow, then you can add more instances to a running process.
One important difference between core and task instance groups is that the core instances store actual data on HDFS whereas task instances do not. In turn, you can only increase the core instance group (because removing running instances would lose the data on those instances). On the other hand, you can both increase and decrease the task instance group by adding or removing task instances.
So these two types of instances can be used to adjust the processing power of your job. Typically, you use ondemand instances for core instances because they must be running all the time and cannot be lost, and you use spot instances for task instances because losing task instances do not kill the entire job (e.g., the tasks not finished by task instances will be rerun on core instances). This is one way to run a large cluster cost-effectively by using spot instances.
The general description of each instance type is available here:
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/InstanceGroups.html
Also, this video may be useful for using EMR effectively:
https://www.youtube.com/watch?v=a5D_bs7E3uc