xargs not splitting on whitespace from aws cli output - bash

This commands returns all the AWS regions separated by whitespace:
aws ec2 describe-regions --query 'Regions[*].RegionName' --output text
eu-north-1 ap-south-1 eu-west-3 eu-west-2 eu-west-1 ap-northeast-2 ap-northeast-1 sa-east-1 ca-central-1 ap-southeast-1 ap-southeast-2 eu-central-1 us-east-1 us-east-2 us-west-1 us-west-2
I'm trying to pipe this to xargs but it's seeing it as a single string:
aws ec2 describe-regions --query 'Regions[*].RegionName' --output text | gxargs -I {} aws cloudformation list-stacks --region {}
Invalid endpoint: https://cloudformation.eu-north-1 ap-south-1 eu-west-3 eu-west-2 eu-west-1 ap-northeast-2 ap-northeast-1 sa-east-1 ca-central-1 ap-southeast-1 ap-southeast-2 eu-central-1 us-east-1 us-east-2 us-west-1 us-west-2.amazonaws.com
gxargs: aws: exited with status 255; aborting
gxargs is just gnu xargs (I'm on Mac).
Also, tried this to use jmespath to create a string from an array with a specific delimiter (which I could use with xargs):
aws ec2 describe-regions --query 'Regions[*].join(",",#.RegionName)'
In function join(), invalid type for value: None, expected one of: ['string'], received: "null"
EDIT: just following up, this is what I wound up with. It insists on throwing an error when it doesn't find a stack- probably the same for other aws cli commands
aws ec2 describe-regions --query 'Regions[*].RegionName' --output text | gxargs -n 1 sh -c 'aws cloudformation describe-stacks --stack-name findme --region $0 || true'

Here's man xargs for -I:
-I replace-str
Replace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks
do not terminate input items; instead the separator is the newline character. Implies -x and -L 1.
You can use xargs -n 1 aws cloudformation list-stacks --region instead

Related

how to iterate over aws cli result using bash for loop ? [describe-images]

Goal: find specific AMI's and copy them to another AWS region.
using describe-images and its filter i get a list of ImageId and Name,
AMI_LIST=$(aws ec2 describe-images --filters "Name=tag:Name,Values=*one*,*two*,*three*,*four*" \
"Name=state,Values=available" "Name=tag:Name,Values=${CUSTOMER_NAME}*" \
--query 'Images[*].{ID:ImageId,NAME:Name}' --output text)
echo $AMI_LIST
result:
ami-036ba4ef9fa1d148d big394_one_1 ami-06d13684f11138f1f big394_two_3 ami-0706803a11e21946d big394_two_1 ami-094043f896db39243 big394_two_2 ami-0c11ff60c981c2273 big394_three_1 ami-0d0b30fcc69f30af8 big394_four_1
then i want to copy the images to another AWS region using a loop:
for ami in $AMI_LIST; do
aws ec2 copy-image --source-image-id ${ami[0]} --source-region us-east-1 --region us-west-2 --name ${ami[2]}
done
ofc it does not work because ${ami[0]} and ${ami[1]} has no meaning, but they represent what i would like to achieve.
i did try to play with converting the list to array but without success.
Thanks.
This should achieve what you expected :
aws ec2 describe-images --filters "Name=tag:Name,Values=*one*,*two*,*three*,*four*" \
"Name=state,Values=available" "Name=tag:Name,Values=${CUSTOMER_NAME}*" \
--query 'Images[*].{ID:ImageId,NAME:Name}' --output text \
| while read ami name; do
aws ec2 copy-image --source-image-id $ami --source-region us-east-1\
--region us-west-2 --name $name
done

get subnet id AWS

i am trying to get the subnet ids within a particular VPC and store them in variables
so I can use them in a bash script
aws ec2 describe-subnets --filter "Name=vpc-id,Values=VPCid" --region $REGION --query "Subnets[*].SubnetId" --output text
and this gives something like this
subnet-12345 subnet-78910
(END)
I wonder how I can store them into a variable.
I tried with
SBnet=$(aws ec2 describe-subnets --filter "Name=vpc-id,Values=VPCid" --region $REGION --query "Subnets[*].SubnetId" --output text)
but then I do not know I can access the array/list created.
I tried with
echo $(SBnet[0])
but does not work
I am on MACos usin zsh
You can do this as follows (add your VPC and the region):
#!/bin/bash
SUBNET_IDS=$(aws ec2 describe-subnets --filter "Name=vpc-id,Values=vpc-1234" --query "Subnets[*].SubnetId" --output text)
for SUBNET_ID in $SUBNET_IDS;
do
echo $SUBNET_ID
done
To split the list of subnet IDs into variables, you can do this:
#!/bin/bash
SUBNET_IDS=$(aws ec2 describe-subnets --filter "Name=vpc-id,Values=vpc-1234" --query "Subnets[*].SubnetId" --output text)
IFS=$'\t ' read -r -a subnet_ids <<< $SUBNET_IDS
echo "${subnet_ids[0]}"
echo "${subnet_ids[1]}"
And the individual subnet IDs will be in the subnet_ids array.
you can do as #jarmod suggested and you could also write a query to extract all the subnets tied to all the VPC's in your system in a comma separated output and use it further like this
aws ec2 describe-subnets --query "Subnets[].[SubnetId,VpcId,CidrBlock,AvailabilityZone]" --output text|sed 's/\t/,/g'

passing command from awk to the next command using xargs

I am using AWS EC2 CLI to perform a filter on stopped instances, then create an AMI out of these with the AMI name taken from the instance tag.
aws ec2 describe-instances --output text --profile proj --query 'Reservations[*].[Instances[*].[InstanceId, InstanceType, State.Name, Platform, Placement.AvailabilityZone, PublicIpAddress, PrivateIpAddress,[Tags[?Key==`Name`].Value][0][0]]]' --filter --filters Name=instance-state-name,Values=stopped | awk '{print $1, $8}' | xargs -n2 aws ec2 create-image --profile proj --instance-id {} --name {} --no-reboot
how to let args differentiate the two different parameters from AWK (instnaceid, instance name tag), thereby it can be correctly pumped into the ec2 create-image on the instance-id and --name parameter accordingly
You do not need awk.Using AWS CLI, you are extracting 8 values first and then using awk to extract 2 values from that 8 values. Why? Just extract the 2 values from AWS CLI without using awk.
--query 'Reservations[*].[Instances[*].[InstanceId, [Tags[?Key==`Name`].Value][0][0]]]'
will return only the values you are interested in. Then use xargs to pass the arguments to your next command.
xargs -n2 command --arg1 $1 --arg2 $2
Your entire command becomes:
aws ec2 describe-instances --output text --profile proj --query 'Reservations[*].[Instances[*].[InstanceId, [Tags[?Key==`Name`].Value][0][0]]]' --filter --filters Name=instance-state-name,Values=stopped | xargs -n2 aws ec2 create-image --profile proj --instance-id $1 --name $2 --no-reboot

Shell script syntax, escape character

I have a shell script as given below. This script actually add AWS instance in autoscalling scale in protection group. When I run individual commands that went fine. But when I created a shell file and tried to execute same there are error. See below script
set -x
INSTANCE_ID=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)
ASG_NAME=$(aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" --region us-east-2 | jq '.Tags[] | select(.["Key"] | contains("a:autoscaling:groupName")) | .Value')
ASG_NAME=$(echo $ASG_NAME | tr -d '"')
aws autoscaling set-instance-protection --instance-ids $INSTANCE_ID --auto-scaling-group-name $ASG_NAME --protected-from-scale-in --region us-east-2
error is as given below. I think issue is with second line. It is not able to get ASG_NAME, I tried some of escape character but nothing is working.
+++ wget -q -O - http://169.254.169.254/latest/meta-data/instance-id
++ INSTANCE_ID=i-----
+++ aws ec2 describe-tags --filters Name=resource-id,Values=i------ --region us-east-2
+++ jq '.Tags[] | select(.["Key"] | contains("a:autoscaling:groupName")) | .Value'
++ ASG_NAME=
+++ echo
+++ tr -d '"'
++ ASG_NAME=
++ aws autoscaling set-instance-protection --instance-ids i---- --auto-scaling-group-name --protected-from-scale-in --region us-east-2
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:
aws help
aws <command> help
aws <command> <subcommand> help
aws: error: argument --auto-scaling-group-name: expected one argument
> Blockquote
Solved issue by recommendation of #chepner. Modified second line by
ASG_NAME=$(aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" --region us-east-2 --query 'Tags[1].Value')

Trimming bash string for aws ecr repo

I need to trim a single " from a bash string both from starting and ending. I tried many things, but still didn't get the output.
Note: I tried $a{// \"}, but it didn't work.
The following code is what I have tried:
repoUri=$(aws ecr create-repository --repository-name $reponame | jq ".repository.repositoryUri")
$repoUri
You could use the -r jq option for "raw output" to suppress the double quotes:
aws ecr create-repository --repository-name "$reponame" \
| jq -r '.repository.repositoryUri'
But you don't actually need jq at all – you can use the --query option in the request, and suppress the double quotes with --output text:
aws ecr create-repository --repository-name "$reponame" \
--query 'repository.repositoryUri' --output text

Resources