jq key, value for each shell script - shell

I'm trying to use every Key,Value of an output and pipe it to another command.
Here is what I'm trying to use:
INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID"
With the above command, I have the following output:
{
"Tags": [
{
"ResourceType": "instance",
"ResourceId": "i-0342a609edf80001a",
"Value": "A-VALUE",
"Key": "A-KEY"
},
{
"ResourceType": "instance",
"ResourceId": "i-0342a609edf80001a",
"Value": "B-VALUE",
"Key": "B-KEY"
},
{
"ResourceType": "instance",
"ResourceId": "i-0342a609edf80001a",
"Value": "C-VALUE",
"Key": "C-KEY"
},
{
"ResourceType": "instance",
"ResourceId": "i-0342a609edf80001a",
"Value": "D-VALUE",
"Key": "D-KEY"
},
{
"ResourceType": "instance",
"ResourceId": "i-0342a609edf80001a",
"Value": "E-VALUE",
"Key": "E-KEY"
},
{
"ResourceType": "instance",
"ResourceId": "i-0342a609edf80001a",
"Value": "F-VALUE",
"Key": "G-KEY"
},
{
Now I want to pipe each Key,Value to the following command:
aws ec2 create-tags --resources XXXXX --tags Key=H-KEY,Value=H-VALUE
Where the quantity and values of Key,Value are variable. So I believe I need a "for each".
May you help me?
It's like: For each Key,Value, do:
aws ec2 create-tags --resources XXXXX --tags Key=A-KEY,Value=A-VALUE
aws ec2 create-tags --resources XXXXX --tags Key=B-KEY,Value=B-VALUE
aws ec2 create-tags --resources XXXXX --tags Key=C-KEY,Value=C-VALUE
aws ec2 create-tags --resources XXXXX --tags Key=N...-KEY,Value=N...-VALUE

jq has a #sh directive to output values properly quoted for the shell:
aws ec2 describe-tags --filters "Name=resource-id,Values=$INSTANCE_ID" \
| jq -r '.Tags[] | #sh "aws ec2 create-tags --resources XXXXX --tags Key=\(.Key),Value=\(.Value)"'
Given your input, this outputs
aws ec2 create-tags --resources XXXXX --tags Key='A-KEY',Value='A-VALUE'
aws ec2 create-tags --resources XXXXX --tags Key='B-KEY',Value='B-VALUE'
aws ec2 create-tags --resources XXXXX --tags Key='C-KEY',Value='C-VALUE'
aws ec2 create-tags --resources XXXXX --tags Key='D-KEY',Value='D-VALUE'
aws ec2 create-tags --resources XXXXX --tags Key='E-KEY',Value='E-VALUE'
aws ec2 create-tags --resources XXXXX --tags Key='G-KEY',Value='F-VALUE'
To execute those as commands pipe into sh:
aws ec2 describe-tags ... | jq -r ... | sh
jq is quite an adventure. You need to add a "select" filter to remove keys that start with "aws:"
jq -r '
.Tags[] |
select(.Key | test("^aws:") | not) |
#sh "aws ... --tags Key=\(.Key),Value=\(.Value)"
'

Related

Parse json with jq in aws cli

i need an output similar to this
{
"InstanceType": "c4.xlarge",
"PrivateIpAddress": "10.54.130.52",
"PlatformDetails": "Windows BYOL",
"State":
"Name": "running"
}
}
Reading the documentation of the jq command I have reached the next output:
aws ec2 describe-instances --instance-ids i-0079e143722b0b8f9 | jq -r '.Reservations[].Instances[] | {InstanceType, PrivateIpAddress, PlatformDetails, State}'
{
"InstanceType": "c4.xlarge",
"PrivateIpAddress": "10.54.130.52",
"PlatformDetails": "Windows BYOL",
"State": {
"Code": 16,
"Name": "running"
}
}
Can anyone explain how to do that?
Regards,
This should work:
aws ec2 describe-instances --instance-ids i-0079e143722b0b8f9 | jq -r '.Reservations[].Instances[] | {InstanceType, PrivateIpAddress, PlatformDetails, State: {Name:.State.Name} }'

Setting SSM parameter as an Environment variable for EC2 - Does not work

I am trying to get and export an SSM parameter as an environment variable to an EC2 using the UserData section of Cloudformation.
The script is trying to append for e.g export WHATS_HER_NAME=Sherlyn to the /etc/profile file. But all i see in the /etc/profile is export WHATS_HER_NAME=. The value is not present. I use amazon linux 2 ami.
here is my cloudformation template.
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"Ec2Instance": {
"Type": "AWS::EC2::Instance",
"Properties": {
"IamInstanceProfile": {
"Ref": "Ec2instanceProfileTest"
},
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"\n",
[
"#!/bin/bash -xe",
"yum update -y aws-cfn-bootstrap",
{
"Fn::Sub": "/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource Ec2Instance --configsets default --region ${AWS::Region}"
},
{
"Fn::Sub": "/opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource EC2Instance --region ${AWS::Region}"
},
{
"Fn::Sub": "echo \"export WHATS_HER_NAME=$(aws ssm get-parameter --name WhatsHerName --region ${AWS::Region} --query 'Parameter.Value')\" >> /etc/profile"
}
]
]
}
}
}
},
"GetSSMParameterPolicy": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName": "GetSsmProperty",
"PolicyDocument": {
"Statement": [
{
"Effect": "Allow",
"Resource": "arn:aws:ssm:ap-southeast-2:012345678901:parameter/WhatsHerName",
"Action": [
"ssm:GetParameters",
"ssm:GetParameter"
]
},
{
"Effect": "Allow",
"Resource": "*",
"Action": [
"ssm:DescribeParameters"
]
}
]
},
"Roles": [
{
"Ref": "InstanceRole"
}
]
}
},
"InstanceRole": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": [
"ec2.amazonaws.com"
]
},
"Action": [
"sts:AssumeRole"
]
}
]
},
"Path": "/"
}
},
"BasicParameter": {
"Type": "AWS::SSM::Parameter",
"Properties": {
"Name": "WhatsHerName",
"Type": "String",
"Value": "Sherlyn"
}
}
}
}
any help would be highly appreciated.
I am not a fan of using JSON for CloudFormation templates so I cannot offer the solution in JSON, but here it is in YAML.
UserData:
Fn::Base64: !Sub
- |
#!/bin/bash -xe
yum update -y aws-cfn-bootstrap
/opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource Ec2Instance --configsets default --region ${AWS::Region}
/opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource EC2Instance --region ${AWS::Region}
echo export WHATS_HER_NAME=${WhatsHerNameParameter} >> /etc/profile
- WhatsHerNameParameter: {{resolve:ssm:WhatsHerName:1}}
You can read more about using AWS Systems Manager Parameter Store Secure String parameters in AWS CloudFormation templates
The snippet above is substituting ${AWS::StackName} and ${AWS::Region} and when it gets to ${WhatsHerNameParameter} it checks for the SSM parameter and substitutes that into the UserData.
This mean that the UserData is complete before it gets to the EC2 instance.
I see two issues:
Your instance doesn't depend on the parameter so the parameter can be created after the instance. When the instance is trying to read the empty parameter, it's just not there. Use DependsOn: [ BasicParameter ].
You didn't include Ec2instanceProfileTest in your sample code. Are you sure it properly uses GetSSMParameterPolicy? If you run that aws ssm get-parameter command after the stack is done, can you get the value properly? If you can't, there might be a permission error. Check the result.

aws ecs update-service not working desired-count =1 but running =0 after jenkins build

What is working fine
I have successfully created a AWS ECS cluster using terraform script
with following parameters
max_instance_size = 2
min_instance_size = 1
desired_capacity = 1
"maximumPercent": 100,
"minimumHealthyPercent": 0
Everything is working fine when I am creating this Cluster . My application is up and running and accessible through Load Balancer.
What is giving problem
Now i have a Jenkins Job to do following steps
Checkout
Build Application
Create Docker Image
Push Docker image into Hub
Deploy the image through Task Definition update.
Here is the Jenkins snippet
stage("Deploy") {
sh "sed -e 's;%BUILD_TAG%;${BUILD_NUMBER};g' accountupdateecs-task-defination.json > accountupdateecs-task-defination-${BUILD_NUMBER}.json"
def currTaskDef = sh (returnStdout: true,script: "aws ecs describe-task-definition --task-definition ${taskFamily}| egrep 'revision'| tr ',' ' '| awk '{print \$2}'").trim()
def currentTask = sh (returnStdout: true,script: "aws ecs list-tasks --cluster ${clusterName} --family ${taskFamily} --output text | egrep 'TASKARNS' | awk '{print \$2}' ").trim()
if(currTaskDef) {sh "aws ecs update-service --cluster ${clusterName} --service ${serviceName} --task-definition ${taskFamily}:${currTaskDef} --desired-count 0 "}
if (currentTask) {sh "aws ecs stop-task --cluster ${clusterName} --task ${currentTask}"}
sh "aws ecs register-task-definition --family ${taskFamily} --cli-input-json ${taskDefile}"
def taskRevision = sh (returnStdout: true, script: "aws ecs describe-task-definition --task-definition ${taskFamily} | egrep 'revision' | tr ',' ' ' | awk '{print \$2}'").trim()
sh "aws ecs update-service --force-new-deployment --cluster ${clusterName} --service ${serviceName} --task-definition ${taskFamily}:${taskRevision} --desired-count 1"
}
Issue
After successfully execution of job my always i can see in cluster
desired-count = 1
running = 0
and application is not available.
Here is the success log of jenkins
+ aws ecs update-service --force-new-deployment --cluster FinanceManagerCluster --service financemanager-ecs-service --task-definition accountupdateapp:20 --desired-count 1
{
"service": {
"serviceArn": "arn:aws:ecs:us-east-1:3432423423423:service/financemanager-ecs-service",
"serviceName": "financemanager-ecs-service",
"clusterArn": "arn:aws:ecs:us-east-1:3432423423423:cluster/FinanceManagerCluster",
"loadBalancers": [
{
"targetGroupArn": "arn:aws:elasticloadbalancing:us-east-1:3432423423423:targetgroup/ecs-target-group/ed44ae00d0de463d",
"containerName": "accountupdateapp",
"containerPort": 8181
}
],
"status": "ACTIVE",
"desiredCount": 1,
"runningCount": 0,
"pendingCount": 0,
"launchType": "EC2",
"taskDefinition": "arn:aws:ecs:us-east-1:3432423423423:task-definition/accountupdateapp:20",
"deploymentConfiguration": {
"maximumPercent": 100,
"minimumHealthyPercent": 0
},
"deployments": [
{
"id": "ecs-svc/9223370480222949120",
"status": "PRIMARY",
"taskDefinition": "arn:aws:ecs:us-east-1:3432423423423:task-definition/accountupdateapp:20",
"desiredCount": 1,
"pendingCount": 0,
"runningCount": 0,
"createdAt": 1556631826.687,
"updatedAt": 1556631826.687,
"launchType": "EC2"
},
{
"id": "ecs-svc/9223370480223135598",
"status": "ACTIVE",
"taskDefinition": "arn:aws:ecs:us-east-1:3432423423423:task-definition/accountupdateapp:19",
"desiredCount": 0,
"pendingCount": 0,
"runningCount": 0,
"createdAt": 1556631640.195,
"updatedAt": 1556631823.692,
"launchType": "EC2"
}
],
"roleArn": "arn:aws:iam::3432423423423:role/ecs-service-role",
"events": [
{
"id": "967c99cc-5de0-469f-8cdd-adadadad",
"createdAt": 1556631824.549,
"message": "(service financemanager-ecs-service) has begun draining connections on 1 tasks."
},
{
"id": "c4d99570-408a-4ab7-9790-adadadad",
"createdAt": 1556631824.543,
"message": "(service financemanager-ecs-service) deregistered 1 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:3432423423423:targetgroup/ecs-target-group/ed44ae00d0de463d)"
},
{
"id": "bcafa237-598f-4c1d-97e9-adadadad",
"createdAt": 1556631679.467,
"message": "(service financemanager-ecs-service) has reached a steady state."
},
{
"id": "51437232-ed5f-4dbb-b09f-adadadad",
"createdAt": 1556631658.185,
"message": "(service financemanager-ecs-service) registered 1 targets in (target-group arn:aws:elasticloadbalancing:us-east-1:3432423423423:targetgroup/ecs-target-group/ed44ae00d0de463d)"
},
{
"id": "c42ee5c9-db5b-473a-b3ca-adadadad",
"createdAt": 1556631645.944,
"message": "(service financemanager-ecs-service) has started 1 tasks: (task fc04530a-479d-4385-9856-adadadad)."
}
],
"createdAt": 1556631640.195,
"placementConstraints": [],
"placementStrategy": [],
"healthCheckGracePeriodSeconds": 0
}
Need some help to understand and resolve this issue
Thanks in advance.

How to delete a consul service

I've created a bunch of test services in my consul cluster I wish to remove, I have tried using the /v1/agent/service/deregister/{service id} - and ensured it runs fine on each node - I can see this run on each node
[INFO] agent: Deregistered service 'ci'
Is there another way to manually clean out these old services ?
Thanks,
Try this
$ curl \
--request PUT \
https://consul.rocks/v1/agent/service/deregister/my-service-id
fetch service info curl $CONSUL_AGETNT_ADDR:8500/v1/catalog/service/$SERVICE_NAME | python -mjson.tool :
{
"Address": "10.0.1.2",
"CreateIndex": 30242768,
"Datacenter": "",
"ID": "",
"ModifyIndex": 30550079,
"Node": "log-0",
"NodeMeta": null,
"ServiceAddress": "",
"ServiceEnableTagOverride": false,
"ServiceID": "log",
"ServiceName": "log",
"ServicePort": 9200,
"ServiceTags": [
"log"
],
"TaggedAddresses": null
},
...
prepare a json file, fulfill the values with the above outputs cat > data.json :
{
"Datacenter": "",
"Node": "log-0",
"ServiceID": "log-0"
}
deregister with: curl -X PUT -d #data.json $CONSUL_AGETNT_ADDR:8500/v1/catalog/deregister
Login in to the consul machine,and issue the command as follow:
consul services deregister -id={Your Service Id}
You can clear service config in config directory mannually

Cloud Formation Wait Condition Timeout but Script executes successfully + Sends CFN Signal

I want to setup an EC2 with ubuntu. After the EC2 is setup a script is run which installs several packages. The log (/var/log/cloud-init-output.log) show that the script does not error and sends a successful finishing signal.
Log
+ /usr/local/bin/cfn-signal -e 0 -r 'AWS CodeDeploy Agent setup complete.' 'https://cloudformation-waitcondition-eu-central-1.s3.eu-central-1.amazonaws.com/arn%3Aaws%3Acloudformation%3Aeu-central-1%XXXXXXXXX%3Astack/XXXXXXXXXX/XXXXXXXX-XXXX-11e6-9ec8-50a68ad4f262/WaitHandle?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20160608T172814Z&X-Amz-SignedHeaders=host&X-Amz-Expires=86399&X-Amz-Credential=XXXXXXXXXXXXXX%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Signature=XXXXXXXXXXXXXXX'
/usr/local/lib/python2.7/dist-packages/cfnbootstrap/packages/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
CloudFormation signaled successfully with SUCCESS.
Cloud-init v. 0.7.5 finished at Wed, 08 Jun 2016 17:34:00 +0000. Datasource DataSourceEc2. Up 99.87 seconds
PS: I added some `X, there are real alphanumeric chars in the link.
Code Script
"UserData": {
"Fn::Base64": {
"Fn::Join": [
"",
[
"#!/bin/bash -ex\n",
"apt-get update\n",
"apt-get -y install python-pip\n",
"apt-get -y install python-setuptools\n",
"apt-get -y install ruby2.0\n",
"apt-get install -y awscli\n",
"pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n",
"# Helper function.\n",
"function error_exit\n",
"{\n",
"/usr/local/bin/cfn-signal -e 1 -r \"$1\" '", { "Ref": "WaitHandle" }, "'\n",
"exit 1\n",
"}\n",
"#Install nodejs, npm\n",
"curl -sL https://deb.nodesource.com/setup_6.x\n",
"apt-get install -y nodejs npm || error_exit 'Failed to install nodejs.'\n",
"npm install pm2 -g || error_exit 'Failed to install pm2.'\n",
"# Install the AWS CodeDeploy Agent.\n",
"cd /home/ubuntu/\n",
"aws s3 --region eu-west-1 cp 's3://aws-codedeploy-eu-west-1/latest/install' . || error_exit 'Failed to download AWS CodeDeploy Agent.'\n",
"chmod +x ./install\n",
"./install auto\n",
"/usr/local/bin/cfn-init --stack ", { "Ref":"AWS::StackId" }, " --resource LinuxEC2Instance", " --region ", { "Ref": "AWS::Region" }, "\n",
"# All is well, so signal success.\n",
"/usr/local/bin/cfn-signal -e 0 -r \"AWS CodeDeploy Agent setup complete.\" '", { "Ref": "WaitHandle" }, "'\n"
]
]
}
},
Code Wait Condition
"WaitHandle": {
"Type": "AWS::CloudFormation::WaitConditionHandle",
"Metadata": {
"AWS::CloudFormation::Designer": {
"id": "761ddc9a-7c3b-41ca-9fa1-21429046b271"
}
}
},
"WaitCondition": {
"Type": "AWS::CloudFormation::WaitCondition",
"Properties": {
"Count": {
"Ref": "InstanceCount"
},
"Handle": {
"Ref": "WaitHandle"
},
"Timeout": "300"
},
"Metadata": {
"AWS::CloudFormation::Designer": {
"id": "a2c91b03-2c8c-4bd5-9c44-efdb89cf5375"
}
}
},
Question
What do I need to change in order to get the script passing.
The problem seems not to appear if aws-cfn-bootstrap is not installed with "pip install https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz\n", instead I use now easy install and call the executables directly with e.g. cfn-signal -e 0 -r \"AWS CodeDeploy Agent setup complete.\" '", { "Ref": "WaitHandle" }, "'\n"

Resources