When file is uploaded to s3bucket, I need to invoke an executable on EC2 Instance. The executable will process a long job and invoke some command line executions
So, I want to run an EXE on EC2 Windows instance from AWS Lambda using .Net Core.
After some research, I figured out the prerequisites to do this
SSM Agent installed on EC2 instance
Create an IAM role for EC2:
AmazonSSMMamangementInstanceCore
IAM role for Lambda
AWSLambdaExecute
AmazonEC2ReadOnlyAccess
AmazonSSMFullAccess
AmazonS3FullAccess
Please advice me if there is any better approach to implement this.
Related
I am creating VM using terraform. Now I want to run some command using remote-exec on that VM before it destroyed.
Is there any way to this?
Thanks
I have a EC2 instance that's already launched using ansible ec2 module having user_data(say data1). Stopped the EC2 instance, now I want to modify the user_data(say data2) and start the instance. Giving modified user_data but its not getting reflected on aws.
To summarize, How to modify user_data of stopped aws EC2 using ansible script(with ec2 or ec2_instance) script.
By default, user data scripts and cloud-init directives run only during the boot cycle when you first launch an instance. You can update your configuration to ensure that your user data scripts and cloud-init directives run every time you restart your instance.
User Data for every restart
i have a shell script which need to be installed over 100 Ubuntu instances/servers.What is the best way to install the same script on all instance without logging into each one.
You can use AWS System Manager , according to AWS Documentation :
You can send commands to tens, hundreds, or thousands of instances by
using the targets parameter (the Select Targets by Specifying a
Tag option in the Amazon EC2 console). The targets parameter accepts
a Key,Value combination based on Amazon EC2 tags that you specified
for your instances. When you execute the command, the system locates
and attempts to run the command on all instances that match the
specified tags
You can Target Instance by tag :
aws ssm send-command --document-name name --targets Key=tag:tag_name,Values=tag_value [...]
or
Targeting Instance IDs:
aws ssm send-command --document-name name --targets Key=instanceids,Values=ID1,ID2,ID3 [...]
Read the AWS Documentation for Details.
Thanks
You have several different options when trying to accomplish this task.
Like Kush mentioned, AWS System manager is great, but is a tightly coupled AWS service.
Packer - You could use Packer to create an AMI of the servers, and have the script installed on them, or just executed whatever the script is doing.
Configuration Management.
Ansible/Puppet/Chef. - These tools allow you to manage thousands of servers with only a couple of commands. My preference would be for Ansible, it is light weight, the syntax is only yaml, connects over ssh, and still allows use of placing shell scripts, if need be.
No matter what instance name I choose, whenever I perform the following on an on prem instance:
aws deploy register --instance-name test --tags "Key=Name,Value=test" --region us-west-2 --debug
The following exception is thrown (always):
2016-04-12 11:02:52,625 - MainThread - awscli.errorhandler - DEBUG - HTTP Response Code: 400
ERROR
A client error (InvalidIamUserArnException) occurred when calling the RegisterOnPremisesInstance operation: Iam User ARN
arn:aws:iam::xxx:user/AWS/CodeDeploy/test is not in a valid format
Register the on-premises instance by following the instructions in "Configure Existing On-Premises Instances by Using AWS CodeDeploy" in the AWS CodeDeploy User Guide.
Despite this error, the user gets created on amazon, and I can continue to register the on prem instance with the following:
aws deploy register-on-premises-instance --instance-name test --iam-user-arn arn:aws:iam::xxx:user/test
aws deploy install --override-config --config-file codedeploy.onpremises.yml --region us-west-2 --agent-installer s3://aws-codedeploy-us-west-2/latest/codedeploy-agent.msi
The instance is registered and the user is created, but when deploying to it, I always get "No hosts succeeded". The logs for the codedeploy agent show no errors.
I am not sure whats happening here either since no logs on either end, in codedeploy console or on the on prem machine codedeploy agent. Any ideas?
Please note I am using Windows Embedded Standard 2010 (which is not in the supported list) with the latest version of aws cli but I have successfully deployed to it in the past (with previous version of aws cli).
Figured it out, seems to be broken* if you try and let 'aws deploy register' create IAM user for you. However, if you create the user first (via console or aws cli), then it will work.
You can pass in the option '--iam-user-arn arn:aws:iam::xxx:user/OnPremCodeDeploy' with the 'aws deploy register' command afterwards.
I created the on prem yml manually with the correct access keys from manually creating user and then finally ran:
aws deploy install --overide-config --config-file conf.onpremises.yml --region us-west-2 --agent-installer s3://aws-codedeploy-us-west-2/latest/codedeploy-agent.msi
* at least w/ codedeployagent OFFICIAL_1.0.1.950_msi and windows embedded
Could you check if the IAM user you registered the on-premises instance with CodeDeploy has proper permissions? Including the following.
"iam:CreateAccessKey",
"iam:CreateUser",
"iam:DeleteAccessKey",
"iam:DeleteUser",
"iam:DeleteUserPolicy",
"iam:ListAccessKeys",
"iam:ListUserPolicies",
"iam:PutUserPolicy",
"iam:GetUser"
This can also be referred here: http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-configure-on-premises-host.html#how-to-configure-on-premises-host-prerequisites
Can anyone please tell me how to run a shell script on all the ec2 instances that are part of an auto scaling group?
The scenario is that I have a script that I want to run on many ec2 instances that are turned on automatically as part of auto scaling group. The native approach is to SSH to each instance and run the script. I am looking for a way by which it can run automatically on all the instances when I run it on one of the ec2 instance or any better way of doing this?
Thanks in advance.
You'll want to add that shell script to the userdata in a NEW Launch Config and then update the autoscaling group.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-shell-scripts
Updating the Launch Config
if you want to change the launch configuration for your Auto Scaling group, you must create a launch configuration and then update your Auto Scaling group with the new launch configuration. When you change the launch configuration for your Auto Scaling group, any new instances are launched using the new configuration parameters, but existing instances are not affected.
https://docs.aws.amazon.com/autoscaling/latest/userguide/LaunchConfiguration.html
You can implement in many different ways...
Use awscli to get all instances in auto scaling group
aws autoscaling describe-auto-scaling-groups --auto-scaling-group-name myTestGroup
SSHkit is interesting tool run commands on remote servers
or
Invest your time to automate your infrastructure in proper tools like puppet, chef. With puppet mcollective you can do magic, including what you have asked about.
Update:
When you add instance to autoscaling group new tag name=aws:autoscaling:groupName, value=name_of_assigned_autoscaling_group is added, thus its easy to find it searching for this tag.
$ asgName=testASG
$ aws ec2 describe-instances --filters Name=tag-key,Values='aws:autoscaling:groupName' Name=tag-value,Values=$asgName --output text --query 'Reservations[*].Instances[*].[InstanceId,PublicIpAddress]'
The output you will get from the command above is the instance name and public IP:
i-4c42aabc 52.1.x.y
You can use this is your script...
I would do it using Chef. Opsworks (an aws product) is chef plus a lot of things that will do exactly what you want and give you even more flexibility.
Run Command perhaps?
You can use it to invoke scripts as well:
*Nice cheat: Select Run Command on the left-side menu of the EC2 section. Click the "Run Command" button. Then setup your AWS-RunShellScript. Type in your code. Then at the bottom there is a dropdown labelled: "AWS Command Line Interface command", select the correct platform and copy/paste the command into a script.
$Command_Id = aws ssm send-command --document-name "AWS-RunPowerShellScript" --targets '{\"Key\":\"tag:Name\",\"Values\":[\"RunningTests\"]}' --parameters '{\"commands\":[\"Start-Process \\\"C:\\\\path\\\\to\\\\scripts\\\\LOCAL_RUN.ps1\\\"\"]}' --comment "Run Tests Locally" --timeout-seconds 3800 --region us-east-1 --query 'Command.CommandId' --output text | Out-String
Per the question: Use your Auto Scaling Group Name instead of "RunningTests." Or, in the console: In your "Run Command" setup. Select the "Specifiying a Tag" radio button, then "Name" and your Auto Scaling Group.
*Note: The command above is windows powershell, but you can convert your script to Linux/OS or whatever by selecting the correct platform when in the Run Command setup.
**Note: Ensure your User on that instance has the AmazonSSMFullAccess permission setup to run the commands.
***Note:The SSM Agent comes installed on Windows Instances by default. If you are running Linux or something else, you might need to install the SSM Agent.
For a simplistic implementation you can use python fabric (http://www.fabfile.org/). It is a tool to run commands from a local or bashin instance to list of servers.
Here is a repo which has basic scaffolding and examples. Lot of CICD tools are having features to target this requirements, but I found fabric the easiest to implement fo simple setup.
https://github.com/techsemicolon/python-fabric-ec2-aws