How to delete all security groups on Amazon ec2? - amazon-ec2

I've created new EC2 spot requests over the last weeks. A new security group was created for every request. When the spot requests were deleted the security groups were not deleted. I've hit the 100 groups limit and want to delete them. The EC2 interface apparently allows only one deletion at a time, which means I would have to make 300 clicks to delete these groups. Or is there a better way to delete multiple security groups with few clicks or lines of code?

THis would need some basic scripting and AWS SDK. you can do this with pretty much all the SDK provided by AWS.
I would prefer AWS-CLI as i already have it installed and configured. This is what I would do:
list all the SGs with describe-security-groups
Install jq (the Json parser for BASH)
Pull the SG IDs (check this for jq syntax)
Once you have the SG IDs, run delete-security-group by usig a for loop.
This is fairly simple and straight forward way of doing wat you want to do. THis can be done by any of the AWS SDKs.
These are just a couple of commands which can be constructed into a Bash script, provided:
You have aws-cli installed and configured
you have jq installed on your system.
If you already have some other AWS SDK installed, then you are better off with that as java/python/ruby...etc all have their own inbuilt way of parsing JSON/HASH/DataStructure.
Hope this helps.

I think you can do this by combining a command that lists all security groups and one other that deletes them.
If you are using the python boto API (for example) that would be:
import boto
conn = boto.connect_ec2(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
groups = conn.get_all_security_groups()
Which returns (as an example): [SecurityGroup:appserver, SecurityGroup:default, SecurityGroup:vnc, SecurityGroup:webserver]
And then you delete them all:
for group in groups:
conn.delete_security_group(group.split(":")[1])
Edit
You should run the commands on your shell.

These solutions only work if you don't have rules in other security groups that reference the security groups in question. I have a couple scripts that will delete a single security group, including the ingress rules in other security groups. I also handle the special case of ingress rules referencing the AWS ELB default security group. If you have this more complex situation, the solutions above won't delete your security group because of these other rules. My scripts are here (one for ec2-classic and one for VPC based security groups): https://gist.github.com/arpcefxl/2acd7d873b95dbebcd42

private static void delete(List<String> sgs) {
AmazonEC2Client ec2 = new AmazonEC2Client(Credentials.getCredentialsProvider());
ec2.setEndpoint("ec2.us-west-2.amazonaws.com"); // default
for(String sg:sgs) {
System.out.println("DELETING SECURITY GROUP " + sg);
DeleteSecurityGroupRequest delReq = new DeleteSecurityGroupRequest().withGroupName(sg);
try {
ec2.deleteSecurityGroup(delReq);
} catch (Exception e) {
// e.printStackTrace();
}
}
}

Related

grafana ec2-instance filter by tags

I am trying to set up a var template in grafana that would allow me to only show ec2 instances with specific ec2 tags. I did find
ec2_instance_attribute(us-east-1, InstanceId, {"tag:app": ["$application"]})
on a grafana community site and i changed it to
ec2_instance_attribute(us-west-2, InstanceId, {"tag:ENV": ["Prod"]})
The tags based off of my ec2 instance tags is and i keep getting a metric error. I tried removing the brackets and tweaking and still can't figure it out. Does anyone know how to create this using this method or another one i haven't thought of?
Your expression should be working. But I had this error also, due to a missing AWS Policy permission. This might be the case for you as well AWS has a default Policy to read CloudWatch data: "CloudWatchReadOnlyAccess ", but creating my own policy as a copy of "CloudWatchReadOnlyAccess" and adding "ec2:DescribeTags" and "ec2:DescribeInstances" made this work for me.
The above answer might be only applicable when you control your access via Roles with Policies.

Getting Around Terraform's Limitations

I'm trying to setup terraform to handle creation of fine-grained user permissions, and have been able to create:
Cognito User Pools, Identity Pools
IAM Roles, Permissions
What I'm struggling with is how to link them together. I have two types of user:
Standard User
Manager
As such, I have found two ways that I could use to correctly hook up the correct IAM policy upon login:
Method 1 - Create a custom attribute, and Use the "Choose Role With Rules" to set a rule to set an IAM policy based on the attribute
Method 2 - Create Cognito Groups, and link users and the required IAM policy to each group.
The problem, as far as I can see, is that Terraform doesn't currently support either of those cases, so I need to find a work around. So, my question is essentially, how do I get around terraform's lack of support in some areas?
I've seen some projects that use [Ruby, Go, etc.] to make up for some of the limitations, but I don't quite understand where to start and what is the best option for my needs. I haven't been able to find much in Google yet (possibly https://github.com/infrablocks/ruby_terraform). Does anyone have a good guide or resource I could use to get started?
If terraform does not support something you can use the local-exec provisioner to execute commands after resource creation. For example you could use the aws cli to add a custom attribute:
resource "aws_cognito_identity_pool" "main" {
# ...
provisioner "local-exec" {
command = "aws cognito-idp add-custom-attributes --user-pool-id ${aws_cognito_identity_pool.main.id} --custom-attributes <your attributes>"
}
}
local-exec docs

Getting all security groups on EC2 via ansible

so I am running ansible against EC2. I need to filter the security groups by their tags. My approach was to get all the security groups and parse them for the required tags, but I am unable to get all the security groups that are present.
Is there a way to either get all the security groups or filter a security group based on a tag and get it's ID?
you could use aws command line interface to fetch security group info and add filters to choose specific security groups. for example, if i wanted to fetch all security groups that were tagged with a tag value of 'Production', this works:
aws ec2 describe-security-groups --filters Name=tag-value,Values=Production
using the command line like this requires that you've configured the command line with your specific AWS credentials (see http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html)
HTH
If you can use Ansible > 2.3 then use ec2_group_facts
You should be able to filter the sections you want either using the filters option in the module or by using a when statement in the Ansible.

Is there a way to create nodes from cloud formation that can ssh to each other without passwords?

I am creating up an AWS Cloud formation template which sets up a set of nodes which must allow keyless ssh login amongst themselves. i.e. One controller must be able to login to all slaves with its private key. The controllers private key is generated dynamically so I do not have access to be able to hard code it into the User-Data of the Template or pass it as a parameter to the template.
Is there a way in Cloud Formation templates to add the controller's public key to slave nodes' authorized keys files?
Is there some other way to use security groups or IAMS to do what is required?
You have to pas the Public key o the master server to the slave nodes in the form of user-data. Cloudformation does support user-data. You may have to figure out the syntax for the same.
In other words, consider it as a simple bash script which will copy the master servers's public key to the slaves. and then you pass this bash script as suer-data so that it gets executed for the 1st time the instance is created.
You will find tons of goggle searches on above information.
I would approach this problem with IAM machine roles. You can grant specific machines certain AWS rights. IAM roles do not apply to ssh access, but to AWS api calls, like s3 bucket access or creating ec2 instances.
Therefore, a solution might look like:
Create a controller machine role which can write to a particular S3 bucket.
Create a slave machine role which can read from that bucket.
Have the controller create an upload a public key into the bucket.
Since you don't know if the controller is created before the slaves, you'll have to have cloud-init set up a cron job every couple minutes that downloads the key from the bucket if it hasn't done so yet.

How do I assign alarms to scaling policies in scaling group using aws-sdk

I'm using a ruby script to help facilitate managing AutoScaling on AWS EC2. I've managed to create AMIs, create LaunchConfigurations, create ScalingGroups, and associating one another.
I'm trying to clone a scaling group, copying its launch configurations, load balancers, etc. and its scaling policies.
To copy its scaling policy I have this code:
orig_scaling_group.scaling_policies.each do |policy|
props = Hash[%i(adjustment_type scaling_adjustment cooldown min_adjustment_step alarms).map { |s| [s, policy.send(s)] }.reject { |k, v| v.nil? }]
clone_scaling_group.scaling_policies[policy.name].put(props)
end
(scaling_groups are instances of AWS::AutoScaling::Group)
This goes over the original's scaling policies, and extracts the data from it, and sets it as a new policy in the clone scaling group.
This works fine, except that it does not copy the alarms to the new policy. I could not find anyway in which I can programmatically assign an alarm to a scaling policy.
How can I do it?
I could not find anyway in which I can programmatically assign an alarm to a scaling policy.
I will be restricting my response to above statement.
In AWS Core Ruby SDK, under put_metric_alarm-instance_method API, you can use alarm_actions method. From above referred documentation:
Please note that, above information is relevant to AWS CORE SDK Ruby and not the older AWS ruby SDK.
Also for AWS-CLI, the similar settings can be found with put-metric-alarm for --alarm-actions option.
Not sure whether this what you are looking for, but here I can certainly see a programmatic way of assigning an alarm to an Autoscaling policy.
Hope this helps.

Resources