I've got a long list of commands strung together (aws cli, jq, cut, sed) that works for me with which I'm trying to now run within a bash script. I'm having problems with the single and double quotes.
If I ran this manually it would be the following:
aws \
--profile app-prod \
--region us-east-1 ec2 describe-instances \
--filter 'Name=tag:ChefRole,Values=foo_bar' \
'Name=instance-state-name,Values=running' \
| jq -r '.Reservations[].Instances[] | [.InstanceId, .Tags[].Value] | #csv' \
| cut -d ',' -f1 \
| sort > sort.out;sed 's/"//g' sort.out
In my script I'm trying to pass in parameters the region and Values
I've tried this:
awsCLI(){
cmd=$(aws --profile app-prod --region ${region} ec2 describe-instances --filter 'Name=tag:ChefRole,Values='${component}'| jq -r '.Reservations[].Instances[] | [.InstanceId, .Tags[].Value] | #csv' | cut -d ',' -f1 | sort > "${_sorted}";sed 's/"//g' "${_sorted}")
}
Placing in the variable - cmd=$(commands here) has worked for me before in other scripts, but not this time.
I also tried just running as is:
aws --profile app-prod --region ${region} ec2 describe-instances --filter 'Name=tag:ChefRole,Values='${component}'| jq -r '.Reservations[].Instances[] | [.InstanceId, .Tags[].Value] | #csv' | cut -d ',' -f1 | sort > "${_sorted}";sed 's/"//g' ${_sorted}"
Either of these didn't work for me. Please advise on best practice for this. Should I break this out? How would you build this long command?
Thanks you.
(Editor's note: first code block split by \ and indented; A.T.)
I was actually trying to eliminate the eternally long horizontal scroll bars from your question, but then decided to convert it into an answer, maybe it actually solves your problem (if not, just tell, will delete):
awsCLI(){
cmd=$(aws \
--profile tealium-prod \
--region "${region}" ec2 describe-instances \
--filter 'Name=tag:ChefRole,Values='"${component}" \
| jq -r '.Reservations[].Instances[] | [.InstanceId, .Tags[].Value] | #csv' \
| cut -d ',' -f1 \
| sort > "${_sorted}" \
; sed 's/"//g' "${_sorted}" \
)
}
I think the main problem was superfluous single tick around Values='"${component}". In general you can concatenate strings by juxtaposition, all you have to remember is that "" expand the variables, whereas '' do not.
The root cause was non-existent indentation. Look up what the backslash in the end of the line does, it will make your life much easier.
If you have the ability to use Python, the boto3 library is very easy to use. The issue with multiple types of quotation marks becomes moot.
Related
This question already has answers here:
How can I parse a YAML file from a Linux shell script?
(23 answers)
Closed 26 days ago.
Here is our yaml:
network:
ethernets:
ens160:
addresses:
- 10.200.2.11/22
gateway4: 10.200.0.1
nameservers:
addresses:
- 8.8.8.8
- 4.4.4.4
search:
- cybertax.live
version: 2
I want to change the dns severs only.
From:
- 8.8.8.8
- 4.4.4.4
to:
- 10.10.10.10
- 10.10.10.11
How can I do this? Note: we cannot use or install yq so this needs to be done through sed or awk. Also, yes I know, this is not recommended, but its what needs to be done right now.
What I have tried so far:
sed -i '/ addresses:/,/ search:/ s/^/# /' $netplan_yaml
sed -i '/ nameservers:/a\ \ \ \ \ \ \ \ addresses:' $netplan_yaml
for i in ${!asar_dns[#]}; do
sed -i "/ addresses:/a\ \ \ \ \ \ \ \ - ${asar_dns[$i]}" $netplan_yaml
done
But this does three things wrong (that I can see).
It matches between addresses and search including the line wiht addresses and search. I only want what is AFTER addresses, and BEFORE search.
It puts the DNS addresses in the associative array between the older addresses that is commented out anywhere there is an "addresses". I dont want to do that on the commented out line.
i dont like how I have to use \ \ \ \ \ \ would much rather use a .* if possible but also need to use the addresses in the associative array.
I found a very hacky solution but it works. Open for feedback.
netplan_yaml=/etc/netplan/00-installer-config.yaml
baddns="$(sed -n '/.*nameservers:/,/.*search:/p' /etc/netplan/00-installer-config.yaml | grep -v 'nameservers\|addresses' | grep -v 'search' | grep -v 10.200 | awk '{print$2}')"
mapfile -t arr_baddns <<<$baddns
for i in ${arr_baddns[#]}; do
sed -i "/ - $i/s/^/#/g" $netplan_yaml
done
for i in ${!asar_dns[#]}; do
sed -i "/ addresses:/a\ \ \ \ \ \ \ \ - ${asar_dns[$i]}" $netplan_yaml
done
If ed is available/acceptable, with your given input.
Something like.
ed -s file.yaml <<-'EOF'
g/nameservers:/;/addresses:/;/search:/-s/[[:digit:].]\{1,\}/10.10.10.11/\
-s/[[:digit:].]\{1,\}/10.10.10.10/
,p
Q
EOF
Using variables for the values, something like:
#!/bin/sh
dns1=10.10.10.10
dns2=10.10.10.11
ed -s file.yaml <<-EOF
g/nameservers:/;/addresses:/;/search:/-s/[[:digit:].]\{1,\}/$dns2/\\
-s/[[:digit:].]\{1,\}/$dns1/
,p
Q
EOF
Change Q to w if in-place editing is required, ala sed -i
Remove the ,p to silence the output.
I have an application which needs to know how many EC2 instances I have in both the pending and running states. I could run the two following commands and sum them to get the result, but that means my awscli will be making two requests, which is both slow and probably bad practice.
aws ec2 describe-instances \
--filters Name=instance-state-name,Values="running" \
--query 'Reservations[*].Instances[*].[InstanceId]' \
--output text \
| wc -l
aws ec2 describe-instances \
--filters Name=instance-state-name,Values="pending" \
--query 'Reservations[*].Instances[*].[InstanceId]' \
--output text \
| wc -l
Is there a way of combining these two queries into a single one, or another way of getting the total pending + running instances using a single query?
Cheers!
Shorthand syntax with commas:
Values=running,pending
You can add several filter values as follows:
aws ec2 describe-instances \
--filters "Name=instance-state-name,Values=running,pending" \
--query 'Reservations[*].Instances[*].[InstanceId]' \
--output text \
| wc -l
My task is to grab latest version of maven in bash script using curl https://apache.osuosl.org/maven/maven-3/
Output should be: 3.8.5
curl -s "https://apache.osuosl.org/maven/maven-3/" | grep -o "[0-9].[0-9].[0-9]" | tail -1
works for me ty
One way is to
Use grep to only get the lines containing the folder link
Use sed with an regex to get only the version number
Use sort to sort the lines
Use tail -n1 to get the last line
curl -s https://apache.osuosl.org/maven/maven-3/ \
| grep folder \
| gsed -E 's/.*([[:digit:]]\.[[:digit:]]\.[[:digit:]]).*/\1/' \
| sort \
| tail -n1
Output:
3.8.5
The url you're mentioning is an HTML-document and curl is not an HTML-parser, so I'd look into an HTML-parser, like xidel, if I were you:
$ xidel -s "https://apache.osuosl.org/maven/maven-3/" -e '//pre/a[last()]'
3.8.5/
$ xidel -s "https://apache.osuosl.org/maven/maven-3/" -e 'substring-before(//pre/a[last()],"/")'
3.8.5
I would suggest to use the central repository instead one of the apache maven sites because the information on central more maded for machine consumption.
It will be taken into account that the file maven-metadata.xml contains the appropriate information.
There are two options using the <version>..</version> tag and the <latest>..</latest> tag.
MVNVERSION=$(curl https://repo1.maven.org/maven2/org/apache/maven/maven/maven-metadata.xml \
| grep "<version>.*</version>" \
| tr -s " " \
| cut -d ">" -f2 \
| cut -d "<" -f1 \
| sort -V -r \
| head -1)
The second option:
MVNVERSION=$(curl https://repo1.maven.org/maven2/org/apache/maven/maven/maven-metadata.xml \
| grep "<latest>.*</latest>" \
| cut -d ">" -f2 \
| cut -d "<" -f1)
I have the following code
#/bin/bash
set -e
set -x
requestResponse=$(ssh jump.gcp.xxxxxxx """source dev2 spi-dev
kubectl get pods -o json | jq '.items[] |select(.metadata.name[0:3]=="fea")' | jq .status.podIP
2>&1""")
echo $requestResponse
In the above code source dev2 spi-dev means we have moved to spi-dev namespace inside dev2 cluster. kubectl get pods -o json | jq '.items[] |select(.metadata.name[0:3]=="fea")' | jq .status.podIP 2>&1""") means to print ip address of pod starting with fea. If I do manually kubectl command works. I have also tried escaping fea like \"fea\"
These triple quotes """ are not working as you expect.
Try to change it like this:
ssh jump.gcp.xxxxxxx << EOF
source dev2 spi-dev
kubectl get pods -o json | \
jq '.items[] | \
select(.metadata.name[0:3]=="fea")' | \
jq .status.podIP 2>&1
EOF
I want to list all ec2 instances for each account we have on ~/.aws/credentials, but the aws ec2 describe-instances doesn't return anything.
ACC=$(cat ~/.aws/credentials | egrep "\[.*\]" | cut -d '[' -f2 | cut -d ']' -f1 | grep -v default | sort)
FILE="${HOME}/ec2.csv"
echo -e "ACCOUNT;EC2" > ${FILE}
for account in ${ACC}
do
for region in $(aws ec2 describe-regions --profile default --region sa-east-1 --output text | cut -f3)
do
EC2=$(aws ec2 describe-instances --profile ${account} --region ${region} --query "Reservations[*].Instances[*].{PrivateIP:PrivateDnsName,Name:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --output text)
echo -e "${account};${EC2}" >> ${FILE}
done
done
i solved this issue replacing the default profile on the second for.
for region in $(aws ec2 describe-regions --profile ${account} --region sa-east-1 --output text | cut -f3)
This script works, i hope this script can help's.