s3 object not looping as expecting - bash

Bash script was working before
Below script broke after a clean install
AWS Command Line Interface is installed and AWS is configured
When is it not looping through each file?
## variables are set before this point
files=$(/usr/local/bin/aws s3api list-objects --bucket "$bucket" --prefix "$target_zipname" --query "Contents[].{Key: Key}")
##return sample##
# [ { "Key": "fmpbks_2016_08_18_17_08_35.zip" }, { "Key": "fmpbks_2016_08_18_17_14_39.zip" }, { "Key": "fmpbks_2016_08_19_10_54_24.zip" }, { "Key": "fmpbks_2016_08_19_10_55_57.zip" }, { "Key": "fmpbks_2016_08_19_10_56_29.zip" }, { "Key": "fmpbks_2016_08_19_11_00_56.zip" } ]
##
for zip_file in $files
do
echo $zip_file # for testing
delete_path="s3://$bucket/$zip_file"
deleted=$(/usr/local/bin/aws s3 rm $delete_path)
break
done
output:
[
{
"Key":
"fmpbks_2016_08_18_17_08_35.zip"
},
{
"Key":
"fmpbks_2016_08_18_17_14_39.zip"
},
{
"Key":
"fmpbks_2016_08_19_10_54_24.zip"
},
{
"Key":
"fmpbks_2016_08_19_10_55_57.zip"
},
{
"Key":
"fmpbks_2016_08_19_10_56_29.zip"
},
{
"Key":
"fmpbks_2016_08_19_11_00_56.zip"
}
]

The aws s3api list-buckets command returns a JSON object by default. The for loop doesn't understand how to interpret JSON.
Use the --output text option to list each file on a separate line:
files=$(aws s3api list-objects --bucket MY-BUCKET --query 'Contents[*].Key' --output text)
By the way, if your goal is to delete all files within a certain path, you could use:
aws s3 rm s3://MY-BUCKET --exclude "*" --include "MY-PREFIX*" --recursive

Related

How to iterate through list of dictionaries and sort the output based on the key in Bash?

We are getting the output from an API call in the below list of dictionaries format,
[
{
"Key": "/builder-deployer/test2/services/serviceA/ip",
"Val": "10.1.2.1"
},
{
"Key": "/builder-deployer/test2/services/serviceA/port",
"Val": "2"
},
{
"Key": "/builder-deployer/test2/services/serviceA/url",
"Val": "serviceA.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceA/username",
"Val": "jenkins"
},
{
"Key": "/builder-deployer/test2/services/serviceB/ip",
"Val": "10.1.2.2"
},
{
"Key": "/builder-deployer/test2/services/serviceB/port",
"Val": "3"
},
{
"Key": "/builder-deployer/test2/services/serviceB/url",
"Val": "serviceB.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceB/username",
"Val": "jenkins"
},
{
"Key": "/builder-deployer/test2/services/serviceC/ip",
"Val": "10.1.2.2"
},
{
"Key": "/builder-deployer/test2/services/serviceC/port",
"Val": "4"
},
{
"Key": "/builder-deployer/test2/services/serviceC/url",
"Val": "serviceC.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceC/username",
"Val": "jenkins"
}
]
I have a requirement to fetch the unique username and IP combination in the form of username#ip from the list to install some dependencies.
i.e. in the above list of dictionaries, we have 2 unique username#ip combinations. So I need to install packages on jenkins#10.1.2.1, jenkins#10.1.2.2 machines respectively.
I have the below code in a python script that does the same but most of our code is in Bash, we want to convert this also to Bash. Any help would be very helpful
ip=[]
for username in outputs:
for ip in outputs:
if username['Key'].split('/')[-2] == ip['Key'].split('/')[-2] and username['Key'].split('/')[-1] in ['username'] and ip['Key'].split('/')[-1] in ['ip']:
ip.append(username['Val']+'#'+ip['Val'])
print(set(ip))
jq
$ FILE="file.json"
$ paste -d# \
<(jq -r '.[]|select(.Key|contains("/username")).Val' "$FILE") \
<(jq -r '.[]|select(.Key|contains("/ip")).Val' "$FILE") | \
sort -u
jenkins#10.1.2.1
jenkins#10.1.2.2
awk
awk -F\" '/\/ip/{getline;ip=$4}/username/{getline;$0=$4"#"ip;print|"sort -u"}' "$FILE"
jenkins#10.1.2.1
jenkins#10.1.2.2
The suggestion in the comments to leave your solution in python (per Gordon Davisson) would be my preference as well. But, if you really have to have a bash solution, the following is one option. Bear in mind it will be fragile if your API JSON format changes over time. It is also not terribly efficient as it relies on reading the source JSON multiple occasions. Further, both grep and sort are also utilized:
#!/bin/bash
input_json="${1:-/tmp/bar/t.json}"
while read -r line ; do
search_str=$(grep -oB 1 "${line}" "${input_json}" \
| awk 'NR==1 {gsub(/^"/,"", $2); \
gsub(/ip",$/,"username", $2); print $2}')
user=$(grep -oA 1 "${search_str}" "${input_json}" | \
awk 'NR==2 {gsub(/"/,"", $2); print $2}')
echo "${user}#${line}"
# get unique ip addresses from source json
done < <(grep -Eo "([0-9]{1,3}[\.]){3}[0-9]{1,3}" "${input_json}" | sort -u)
Sample data:
[
{
"Key": "/builder-deployer/test2/services/serviceA/ip",
"Val": "10.1.2.1"
},
{
"Key": "/builder-deployer/test2/services/serviceA/port",
"Val": "2"
},
{
"Key": "/builder-deployer/test2/services/serviceA/url",
"Val": "serviceA.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceA/username",
"Val": "jenkins"
},
{
"Key": "/builder-deployer/test2/services/serviceB/ip",
"Val": "10.1.2.2"
},
{
"Key": "/builder-deployer/test2/services/serviceB/port",
"Val": "3"
},
{
"Key": "/builder-deployer/test2/services/serviceB/url",
"Val": "serviceB.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceB/username",
"Val": "jenkins"
},
{
"Key": "/builder-deployer/test2/services/serviceC/ip",
"Val": "10.1.2.2"
},
{
"Key": "/builder-deployer/test2/services/serviceC/port",
"Val": "4"
},
{
"Key": "/builder-deployer/test2/services/serviceC/url",
"Val": "serviceC.abc.com"
},
{
"Key": "/builder-deployer/test2/services/serviceC/username",
"Val": "jenkins"
}
]
Sample output:
$ ./t.sh
jenkins#10.1.2.1
jenkins#10.1.2.2
Use jq once, no need of temporary file
$ cat test.json | jq -r '.[]|select(.Key|test("(username|ip)$")).Val' \
| paste -d '#' - - | sort -u
10.1.2.1#jenkins
10.1.2.2#jenkins

Bash get value of matching substring in a long string

I'm working on a bash script and having hard time extracting value from a matching substring.
I cannot share curl command as it has sensitive information but updating i value(1)(updated to mimic real value) that I'm having problem with. jq gives parse error
curl -s -g "$line" | jq -c '.allBuilds[]' | while read i; do
job_name=$(echo "$i" | jq .fullDisplayName | tr -d '»' | tr -s " " | sed 's/ /,/g' | tr -d '"')
done
expected output
Hello,Java,World,master,#47
I get expected output on most of the i values but some error out
Below are sample i values.
1
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.StringParameterValue","name":"environment","value":"DE"},{"_class":"hudson.model.BooleanParameterValue","name":"update","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"black","value":false},{"_class":"hudson.model.StringParameterValue","name":"description","value":"DE"},{"_class":"hudson.model.StringParameterValue","name":"number","value":""},{"_class":"hudson.model.StringParameterValue","name":"config","value":"{ "E": "DE", "Exp": "1111", "Pr": "D", "Man": { "Se": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Se-B8SKMz", "OR": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/OR-kvJ2lJ", "AR": "arn:aws:secretsmanager:region:111111111111:secret:A/DE/SA/rds", "User": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/mUixbWY", "sales": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/saELY", "vau": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/vNRR7BO", "sc": "", "exAd": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/ExyBoYL", "exp": "arn:aws:secretsmanager:region:111111111111:secret:BA/A/Exl67GE", "sec": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Secle06a", "Secu": "arn:aws:secretsmanager:region:111111111111:secret:DE/A/Seia" }, "s3": { "buckets": { "hello": { "name": "helloDEBA", "region": "region", "account": "111111111111" }, "mlt": { "name": "sacdhbd", "region": "region", "account": "111111111111" }, "devo": { "name":"devvvvv", "region": "region", "account": "5555555" } } }, "roles": { "lam": "arn:aws:iam::111111111111:role/lam", "lambd": "arn:aws:iam::111111111111:role/lambd", "la": "arn:aws:iam::111111111111:role/lam", "la": "arn:aws:iam::111111111111:role/la", "la": "arn:aws:iam::111111111111:role/la","lasds": "arn:aws:iam::111111111111:role/lafgg", "lafdg": "arn:aws:iam::111111111111:role/dfsdv", "acc": "arn:aws:iam::111111111111:role/acc" }, "vpc": { "subnets": { "private": { "1a": "subnet-111111", "1b": "subnet-22222", "1c": "subnet-33333" } }, "securityGroupIds": { "lambda": "sg-1111" }, "endpoints": { "e": "" }, "links": { "b": "" } }, "securi": { "level": "FAILURE", "s": true }, "log": "debug", "se": "hello.com", "sa": { "env": "--DE" }, "lam": { "sss": { "environment": { "variables": { "test": "hello.com", "PhoneNumber": "11111" } } } } }"},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"ch","value":false}]},{"_class":"jenkins.scm.A.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello » Java » World » master #25","id":"25","number":25,"timestamp":1575582153372}
2
{"_class":"org.jenkinsci.plugins.workflow.job.WorkflowRun","actions":[{"_class":"hudson.model.CauseAction"},{"_class":"hudson.model.ParametersAction","parameters":[{"_class":"hudson.model.BooleanParameterValue","name":"helo-world","value":false},{"_class":"hudson.model.StringParameterValue","name":"environment","value":"hello"},{"_class":"hudson.model.StringParameterValue","name":"config","value":""},{"_class":"hudson.model.StringParameterValue","name":"description","value":""},{"_class":"hudson.model.BooleanParameterValue","name":"hello","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"hello2","value":false},{"_class":"hudson.model.BooleanParameterValue","name":"scan","value":false},{"_class":"hudson.model.StringParameterValue","name":"hello3","value":""}]},{"_class":"jenkins.scm.api.SCMRevisionAction"},{},{"_class":"hudson.plugins.git.util.BuildData"},{"_class":"hudson.plugins.git.GitTagAction"},{},{},{},{"_class":"org.jenkinsci.plugins.workflow.cps.EnvActionImpl"},{"_class":"hudson.plugins.git.util.BuildData"},{},{},{},{},{},{"_class":"org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction"},{},{"_class":"org.jenkinsci.plugins.workflow.job.views.FlowGraphAction"},{},{},{},{}],"fullDisplayName":"Hello»Java»World»master#21","id":"21","number":21,"timestamp":1574705384077}
I'm trying to extract fullDisplayName value and I have tried some options like
printf '%s\n' "$i" | grep fullDisplayName
printf '%s\n' gives output in new lines when tried in shell but in script its different behavior
{}
{}
{}]
fullDisplayName:Hello » Java » World » master #25
id:25
number:25
timestamp:1575582153372
It would appear that you should be using read -r.
It also appears that it would be much simpler if you focused on using curl and jq to extract the information, without any grep or tr invocation and without any shell looping. Assuming you can arrange for the output of curl to be valid JSON(*), a single invocation of jq along the following lines should do the job:
jq -c '.allBuilds[] | .fullDisplayName | gsub("»";"")'
(*) To check whether the output of curl is valid, you can pipe the output of your curl command into jq empty:
curl ... | jq empty

How to filter a value using jq'

I am writing a script to create AWS volumes automatically. I am trying to filter out the value for volumeId using echo and jq
[root#ip-172-41-23-8 ~]# aws ec2 describe-volumes --filters Name=status,Values=available Name=availability-zone,Values=us-east-1d Name=tag:Name,Values=*USEPROD*|jq ".VolumeId"
null
[root#ip-172-41-23-8 ~]# echo $value | jq
{
"Volumes": [
{
"AvailabilityZone": "us-east-1d",
"Attachments": [],
"Tags": [
{
"Value": "Dev",
"Key": "Env"
},
{
"Value": "Shell",
"Key": "Deployment"
},
{
"Value": "cnx30au - Kafka-Data-Node - /dev/sdg - /abc/data/sda",
"Key": "Name"
}
],
"Encrypted": true,
"VolumeType": "gp2",
"VolumeId": "vol-095ac513f0f049ba5",
"State": "available",
"KmsKeyId": "arn:aws:kms:us-east-1:709721360603:key/xvfgfr-dsdsd-dsdsds-a00c-dsdsdsd34dsd",
"SnapshotId": "",
"Iops": 100,
"CreateTime": "2019-04-08T21:52:31.473Z",
"Size": 10
}
]
}
json="aws ec2 describe-volumes --filters Name=status,Values=available Name=availability-zone,Values=us-east-1d Name=tag:Name,Values=*USEPROD*"
echo "Executing: ${json}"
value=`${json}`
valueid=`echo ${value}|jq '.VolumeId'`
[root#ip-172-31-87-1 ~]# echo $valueid
null
When i filter for available volumes i only want the output to be as follows
vol-095ac513f0f049ba5
VolumeId is twice nested, you need to identify that with your query
valueid=`echo ${value} | jq '.[]|.[]|.VolumeId'`
Or simply:
jq '.Volumes[].VolumeId'

Creating Route53 Record Sets via Shell Script

As a part of my shell script, I am trying to create record sets in AWS Route53. However, using variables in the aws cli within my shell script to create those records set, my variables exported in the Shell scripted are not being passed to the aws cli command.
AWS CLI command provided by AWS:
$ aws route53 change-resource-record-sets --hosted-zone-id 1234567890ABC \
--change-batch file:///path/to/record.json
I do not want to create a separate json file on my computer for simplicity purpose, I want to have all my commands and variables within the shell script.
#!/bin/bash
export TARGET_ENVIRONMENT=uat
export BASE_ENVIRONMENT_DNS=abcd-External-9982627718-1916763929.us-west-1.elb.amazonaws.com
# Creates route 53 records based on env name
aws route53 change-resource-record-sets --hosted-zone-id 1234567890ABC
--change-batch '{ "Comment": "Testing creating a record set",
"Changes": [ { "Action": "CREATE", "ResourceRecordSet": { "Name":
"$(TARGET_ENVIRONMENT).company.com", "Type": "CNAME", "TTL":
120, "ResourceRecords": [ { "Value": "$(BASE_ENVIRONMENT_DNS)" } ] } } ] }'
This last command is creating a record set on the AWS Route53 as:
$(TARGET_ENVIRONMENT).company.com
with CNAME as
$(BASE_ENVIRONMENT_DNS)
and NOT really as I want it to be, which is:
uat.company.com
with the CNAME:
abcd-External-9982627718-1916763929.us-west-1.elb.amazonaws.com
How can I pass the environment variables into my aws cli command within the script?
Any help will be highly appreciated.
Thanks!
Variables do not expand within single quotes.
If you close the single quotes before your variable expansion, and then open them again immediately after, it should produce the desired effect. You may or may not need to wrap the variable in double quotes for it to expand.
#!/bin/bash
ENV=uat
DNS=abcd-External-9982627718-1916763929.us-west-1.elb.amazonaws.com
# Creates route 53 records based on env name
aws route53 change-resource-record-sets \
--hosted-zone-id 1234567890ABC \
--change-batch '
{
"Comment": "Testing creating a record set"
,"Changes": [{
"Action" : "CREATE"
,"ResourceRecordSet" : {
"Name" : "'" $ENV "'.company.com"
,"Type" : "CNAME"
,"TTL" : 120
,"ResourceRecords" : [{
"Value" : "'" $DNS "'"
}]
}
}]
}
'
You could also leverage process substitution to keep the JSON formatted:
#!/bin/bash
export TARGET_ENVIRONMENT=uat
export BASE_ENVIRONMENT_DNS=abcd-External-9982627718-1916763929.us-west-1.elb.amazonaws.com
# Creates route 53 records based on env name
aws route53 change-resource-record-sets --hosted-zone-id 1234567890ABC --change-batch file://<(cat << EOF
{
"Comment": "Testing creating a record set",
"Changes": [
{
"Action": "CREATE",
"ResourceRecordSet": {
"Name": "${TARGET_ENVIRONMENT}.company.com",
"Type": "CNAME",
"TTL": 120,
"ResourceRecords": [
{
"Value": "${BASE_ENVIRONMENT_DNS}"
}
]
}
}
]
}
EOF
)
This worked for me!
#!/bin/bash
zoneid=xxxxyyyyyzzzzz
recordname=mycname
recordvalue=myendpoint
aws route53 change-resource-record-sets \
--hosted-zone-id $zoneid \
--change-batch '
{
"Comment": "Creating a record set for cognito endpoint"
,"Changes": [{
"Action" : "CREATE"
,"ResourceRecordSet" : {
"Name" : "'$recordname'.mydomain.com"
,"Type" : "CNAME"
,"TTL" : 120
,"ResourceRecords" : [{
"Value" : "'$recordvalue'"
}]
}
}]
}
You need to close the single quotation mark. Here is the fixed snippet.
#!/bin/bash
zoneid=xxxxyyyyyzzzzz
recordname=mycname
recordvalue=myendpoint
aws route53 change-resource-record-sets \
--hosted-zone-id $zoneid \
--change-batch '
{
"Comment": "Creating a record set for cognito endpoint"
,"Changes": [{
"Action" : "CREATE"
,"ResourceRecordSet" : {
"Name" : "'$recordname'.mydomain.com"
,"Type" : "CNAME"
,"TTL" : 120
,"ResourceRecords" : [{
"Value" : "'$recordvalue'"
}]
}
}]
}'

Mesos Marathon deploy was failed when added parameters in Json file

I was trying to deploy ngnix docker container by mesos Marathon, I would like to set some environment variables in the container, so I added parameter section in the Json file, but after I added the parameter section, it was failed. My Json file as following:
{
"container":{
"type":"DOCKER",
"docker":{
"image":"nginx",
"network":"BRIDGE",
"portMappings":[{"containerPort":80,"hostPort":0,"servicePort":80,"protocol":"tcp"}],
"parameters": [
{ "key": "myhostname", "value": "a.corp.org" }
]
}
},
"id":"nginx7",
"instances":1,
"cpus":0.25,
"mem":256,
"uris":[]
}
my launch script was: curl -X POST -H "Content-Type: application/json" 10.3.11.11:8080/v2/apps -d#"$#"
The command I ran was: ./launch.sh nginx.json
You used the wrong parameter key myhostname, if you want to setup hostname for you container, it should be:
"parameters": [
{ "key": "hostname", "value": "a.corp.org" }
]
if you want to pass environment variable, it should be:
"parameters": [
{ "key": "env", "value": "myhostname=a.corp.org" }
]

Resources