To launch docker-compose, I use AWS ECS.
When create cluster, ec2 instance automatically launched.
And can't launch with none instance. Must have at least 1 instance.
To time sync(https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html), I have to execute shell script before ec2 instance running.
I searched google few hours, there is no correct answer.
Is there any solution to launch cluster with user data?
Thanks.
Related
I have created my own EC2 instance in AWS. That AMI is AWS ECS optimized AMI for launching ecs service from my EC2 instance. I previously discussed the same thing. And tried with that approach. The link is below,
Microservice Deployment Using AWS ECS Service
I created my cluster and configured that cluster name when I am creating optimized AMI by following code snippet in advanced userdata section,
#!/bin/bash
echo ECS_CLUSTER=your_cluster_name >> /etc/ecs/ecs.config
I followed the documentation of cluster creation from following link,
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create_cluster.htmlecs
But, no result - when creating cluster and ECS task definitions it creates and launches into one EC2. And again creating another EC2 by specifying above code. So total 2 Ec2. I already created my own ECS optimized.
I am finding for launching ECS service from my own AMI (that I created). Actually I need to launch my ECS service from my Ec2 (I had created my machine Amazon optimized AMI).
The reason behind this requirement is I don't want to launch my services in machine that owned by others. I need to launch from my machine. And also I need to host my angular application in the same my machine. So I need control of my machine. How can I do this?
Sounds like you just need to create a Launch Configuration. With this you can specify the User Data settings that should be applied when a host is setup.
After you create your Launch Configuration, create a new Auto Scaling Group based off of it (there's a drop-down to select the launch configuration you want to use).
From here, any new instances launched under that ASG will apply the settings you've configured in the associated Launch Configuration.
I have an Amazon EC2 instance which is registered to a cluster of Amazon ECS.
And I want to change this instance's type from c4.large to c4.8xlarge.
I'm able to change its type from c4.large to c4.8xlarge in AWS console. But after the change, I found
[ERROR] Could not register module="api client" err="ClientException: Container instance type changes are not supported. Container instance XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX was previously registered as c4.large.
being printed in /var/log/ecs/ecs-agent.log.20XX-XX-XX-XX file.
Is it possible to change ec2 instance type and re-register it to a cluster?
I think maybe deregister it first, then register it again should work. But I'm afraid this may cause something irreversible in my AWS working environment. So I haven't tried this method yet.
To solve this connection problem between the agent and cluster, just delete the file /var/lib/ecs/data/ecs_agent_data.json and restart docker and ECS.
After that, a new container instance will be created in your cluster with the new size.
sudo rm /var/lib/ecs/data/ecs_agent_data.json
sudo service docker restart
sudo start ecs
Then you can go to the ECS cluster console and deregister the old container instance
UPDATE:
According to #florins and #MBear commented below, AWS updated the data file on ECS instances.
sudo rm /var/lib/ecs/data/agent.db
sudo service docker restart
sudo start ecs
As of March 2021 / AMI image ami-0db98e57137013b2d, /var/lib/ecs/data/ecs_agent_data.json mentioned in the last useful answer does not exist. For me, the commands to execute on the changed instance were:
sudo rm /var/lib/ecs/data/agent.db
sudo service docker restart
After that, it was possible to deploy containers to the instance, without fresh registration (AWS automatically registered a second ECS container instance of the new type). I did have a leftover container instance with the resources of the old instance type to remove.
You can't do this. Per their docs:
The type of EC2 instance that you choose for your container instances determines the resources available in your cluster. Amazon EC2 provides different instance types, each with different CPU, memory, storage, and networking capacity that you can use to run your tasks. For more information, see Amazon EC2 Instances.
This means that when you launch a container on an instance, the agent gathers a bunch of metadata about the instance to run it. If you change it, all of that metadata (or a lot) has changed in a bad way. CPU units, memory, etc. The agent is aware of this and will report it as an error.
You should spin up a new instance of the new type and register it to the cluster and let the task run on it. If it's a service, just terminate the old instance and let it run it against the new one.
I can't think of any real reason why terminating your old instance would cause something irreversible unless it is misconfigured or fragile via user specific settings, by default this would not cause anything destructive.
As alternative approach if the EC2 instance does not store any valuable a new instance using the old instance as template could be started. This takes all existing values and can be achieved just with a few clicks in minutes.
Select the EC2 instance and then "Actions -> Images and templates -> Start more like this". Just change the instance type.
When the instance is running got the the ECS cluster to the tab "ECS instances" and activate the new created instance.
Shutdown the old instance
Update your task maybe taking more cpu and memory and update the service to take the new task revision
If I have a Amazon Redshift instance and an Amazon EC2 instance (running windows amongst other things), can I set up windows scheduled jobs in the EC2 instance that connects to Redshift and runs copy commands?
Really what I am asking is 'is EC2 just a VM on the cloud' and can I do anything I like in it (like set up windows scheduled jobs and be guaranteed they will run on a scheduled time)
It seems that AWS data pipeline is the recommended way to have scheduled jobs load data into Resdshift but this starts to get pricey with frequent jobs
I ran up a redshift instance and a EC2 windows 2012 instance
I installed the ODBC redshift driver
I ran a VBScript that incremented a counter in a table
I scheduled that script in task scheduler
I logged out of EC2 and came back and the data was updated.
So it seems that using windows scheduler on EC2 is a valid alternative to AWS data pipeline of you want to do it that way.
I haven't yet tried the copy command but I will come back and document that also if I have time
I have setup the hadoop cluster on Amazon EC2 using cloudera manager. Cloudera manager created two instances and all is working as expected. I am trying to stop the cloudera created instances through AWS console but there is no option to stop. We have only "Terminate" and "Reboot". I don't want to terminate these instances as i want to reuse these instances.
How to stop these instances ?
Since your instances came from an instance-store backed AMI you will only be able to reboot and terminate the instances. Look in the Management Console under "root device" to confirm this is the case.
To get around this, you can create an AMI from your instances then restart your environment using the new AMI which would give you the option to stop your instances.
Sometimes my application dies without any reason and I can detect that using CloudWatch and CPU usage metric going down.
At this moment I want to restart the java application or the whole EC2 instance.
Any suggestions how can I achive that?
Thanks.
AWS CloudWatch now provides a reboot EC2 instance action.
You can let CloudWatch terminate your EC2 instance and let AutoScaling bring up another "fresh" instance with your application configured.
If only your application halt but EC2 instance works.
You could write a shell monitor the app using CloudWatch API and shoot the app when necessary,then make it a task in cron. Or you can using a shell right in your EC2 instance.
To restart your EC2 instance automatically is way too dangerous ,you can try Autoscaling, which automatically starts a new instance when your instance frozen and keep your SLA.
You can configure in Cloudwatch Alarm EC2 Action for reboot the instance.
Screenshot:
References:
https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-action-reboot-ec2-instance/