I am getting a list of quoted strings back from AWS,
and I would like to isolate one of them into a parameter.
the command I am using is:
allelb=$(aws elb describe-load-balancers --query 'LoadBalancerDescriptions[].LoadBalancerName') && echo $allelb
And this is the output I am getting:
[ "elb-app-mprest-dev", "elb-core-mprest-dev", "api-vector-k8s-local-0j8ccl", "a2e6a899d111011e897b0067693cf815", "api-clusters-sydney7-mpre-rqae1h" ]
What I want to do is, get only the string with the word 'sydney7' inside a parameter. So I will have a new parameter with this content inside :
ELB=api-clusters-sydney7-mpre-rqae1h
This is waht I got so far:
allelb=$(aws elb describe-load-balancers --query 'LoadBalancerDescriptions[].LoadBalancerName' | tr -d '"' | tr -d ',' | tr -d ']' | tr -d '[') && echo $allelb | grep -o sydney7
but the output I get is not good enough-
sydney7
How can I achieve that?
Since the output from your command is a JSON array, you should probably use a program for parsing and filtering JSON, like jq:
$ jq '.[] | match(".*sydney.*").string' <<< "$allelb"
"api-clusters-sydney7-mpre-rqae1h"
If you want it raw, use the -r flag:
$ jq -r '.[] | match(".*sydney.*").string' <<< "$allelb"
api-clusters-sydney7-mpre-rqae1h
With cut
cut -d '"' -f10
with grep
grep -o '[^"]*sydney7[^"]*'
Related
Example here:
gitrepo=$(jq -r '.gitrepo' 0.json)
releasetag=$(curl --silent ""https://api.github.com/repos/\"$gitrepo\""/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
echo "$releasetag"
Used \" to escape characters.
0.json:
{
"type": "github-releases",
"gitrepo": "ipfs/go-ipfs"
}
How to put $gitrepo to work inside $releasetag?
Thanks in advance!
Bash variables expand inside quoted " strings.
gitrepo="$(jq -r '.gitrepo' 0.json)"
releasetag="$(
curl --silent "https://api.github.com/repos/$gitrepo/releases/latest" \
| grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/'
)"
echo "$releasetag"
Btw, as you are using jq to extract .gitrepo from 0.json, you could also use it in the exact same way to extract .tag_name from curl's output (instead of using grep and sed) like so:
gitrepo="$(jq -r '.gitrepo' 0.json)"
releasetag="$(
curl --silent "https://api.github.com/repos/$gitrepo/releases/latest" \
| jq -r '.tag_name'
)"
echo "$releasetag"
And to simplify it even further (depending on your use case), just write:
curl --silent "https://api.github.com/repos/$(jq -r '.gitrepo' 0.json)/releases/latest" \
| jq -r '.tag_name'
I try to make a tool to get list of proxies and You have downloaded Index for one of the free proxy sites I use this :
wget http://free-proxy.cz/en/proxylist/country/all/https/ping/all
and outputs something like that :
<script type="text/javascript">document.write(Base64.decode("MTg1LjExNC4xMzcuMTQ="))</script></td><td style=""><span class="fport" style=''>12195</span></td><td><small>HTTPS</small></td><td class="left"><div style="padding-left:2px"><img src="/flags/blank.gif" class="flag flag-ua" alt="Ukraine" /> Ukraine</div></td><td class="small"><small></small></td><td class="small"><small></small></td><td class="small"><small>High anonymity</small></td><td> <i class="icon-black icon-question-sign"></i></td><td><small>2.4%</small><div class="progress"><div class="fill" style="width:4%;background-color:red;"></div></div></td><td><div style="padding-left:5px"><small>649 ms</small> <div class="progress"><div class="fill" style="width:94%;background-color:#A5DA74;;"></div></div></div></td><td><small>8 hours ago</small></td></tr><tr><td style="text-align:center" class="left"><script type="text/javascript">document.write(Base64.decode("MTYxLjk3LjEzOC4yMzg="))</script></td><td style=""><span class="fport" style=''>3128</span></td><td>
As you can see the IP is encrypted in base64 and port is normal
I try to grep base64 codes first and this is work ↓
echo (outputs) | grep -Eo '("[A-Za-z0-9]{12,30}[=]{0,2}")' | cut -d '"' -f2
and I try this code to get ports ↓
echo (output) | grep -Eo "(class=\"fport\" style=''>[0-9]{1,9})" | cut -d '>' -f2
how can I mix it and make it like that
(base64 code):(port)
and after that I wanna decrypt the base64 code and make it look like :
IP:PORT
1st step
base64 is not an encryption, but an encoding. If you are working on
Linux or other Unix-variants, the command base64, base64-encoder/decoder,
will be pre-installed. If not, it will be easily installed with your
OS-dependent package manager.
Then please try to execute:
base64 -d <<< "MTg1LjExNC4xMzcuMTQ="
It will output:
185.114.137.14
2nd step
Then we can combine the base64 decoder with your command pipeline.
The problem is base64-coding ignores newlines and we need to process
the result of the pipeline line by line. Assuming the variable $output
holds the output of the wget command, please try:
while IFS= read -r line; do
base64 -d <<< "$line"
echo
done < <(echo "$output" | grep -Eo '("[A-Za-z0-9]{12,30}[=]{0,2}")' | cut -d '"' -f2)
It will print something like:
185.114.137.14
161.97.138.238
The <(command) notation is a process substitution and the output of
echo .. grep .. cut pipeline is fed to the while loop via stdin
and the while loop processes the base64-encoded strings line by line.
3rd step
Now we want to merge the IPs and PORTs in a format of IP:PORT.
We can make use of paste command. The final script will be:
paste -d ":" <(
while IFS= read -r line; do
base64 -d <<< "$line"
echo
done < <(echo "$output" | grep -Eo '("[A-Za-z0-9]{12,30}[=]{0,2}")' | cut -d '"' -f2)
) \
<(echo "$output" | grep -Eo "(class=\"fport\" style=''>[0-9]{1,9})" | cut -d '>' -f2)
Output:
185.114.137.14:12195
161.97.138.238:3128
The paste command takes filenames as arguments. Here we make use
of process substitution again in a manner as: paste <(command1) <(command2)
which saves to create temporary files.
I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json)
Postscript
For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:
EMAILID=foo#bar.com # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'
The output is a JSON string:
"foo#bar.com"
I resolved this issue by escaping the inner double quotes
projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
Little unrelated but I will still put it here,
For other practical purposes shell variables can be used as -
value=10
jq '."key" = "'"$value"'"' file.json
Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:
.items[] | select(.name=="string")
in bash you could do
EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')
essentially escaping the quotes and passing it on to jq
It's a quote issue, you need :
projectID=$(
cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)
If you put single quotes to delimit the main string, the shell takes $EMAILID literally.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
Jq now have better way to access environment variables, you can use env.EMAILID:
projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
Another way to accomplish this is with the jq "--arg" flag.
Using the original example:
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] |
select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[]
| select(.username=="$EMAILID") | .id')
echo "$projectID"
See here, which is where I found this solution:
https://github.com/stedolan/jq/issues/626
I know is a bit later to reply, sorry. But that works for me.
export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"
And now I am able to fetch and pass the content of the variable to jq
export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"
In my case I needed to use double quote and quote to access the variable value.
Cheers.
I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:
$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""
a curl command gives me a JSON output as follows:- [{"Description":"","Value":"bc","Key":"a"},{"Description":"","Value":"ef","Key":"d"},...]
Now I want to capture the values of only "Key" & "Value" in such a way that it if I perform echo $a it gives "bc", echo $d it gives "ef" etc.
Note: I am absolutely new in shell scripting/bash, thus this question
Thank you in advance
You can use jq to parse the array and obtain the necessary value.
echo '[{"Description":"","Value":"bc","Key":"a"},{"Description":"","Value":"ef","Key":"d"}]' | jq '.[] | select(.Key == "d").Value'
This Should give you "ef" and,
echo '[{"Description":"","Value":"bc","Key":"a"},{"Description":"","Value":"ef","Key":"d"}]' | jq '.[] | select(.Key == "a").Value'
Will give you "bc"
You can replace the echo with your curl statement.
More about jq : https://stedolan.github.io/jq/
Something like this may do the job :
dataString='[{"Description":"","Value":"bc","Key":"a"},{"Description":"","Value":"ef","Key":"d"},...]'
while read value key; do
echo "key: '$key', value : '$value'"
done < <(echo "$dataString" | sed 's/},{/\n/g' | tr -d '[]{}"' | tr ':' ',' | cut -d',' -f 4,6 --output-delimiter=' ')
Details :
dataString is the output of curl
the whole (echo "$dataString"....) simply turns $dataString into :
bc a
ef d
the <(...) construct is a process substitution . It allows reading the output of a command with the same method you'd be reading from a file
I have written a script to retrieve certain value from file.json. It works if I provide the value to jq select, but the variable doesn't seem to work (or I don't know how to use it).
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
#this does not work *** no value is printed
projectID=$(cat file.json | jq -r '.resource[] | select(.username=="$EMAILID") | .id')
echo "$projectID"
Consider also passing in the shell variable (EMAILID) as a jq variable (here also EMAILID, for the sake of illustration):
projectID=$(jq -r --arg EMAILID "$EMAILID" '
.resource[]
| select(.username==$EMAILID)
| .id' file.json)
Postscript
For the record, another possibility would be to use jq's env function for accessing environment variables. For example, consider this sequence of bash commands:
EMAILID=foo#bar.com # not exported
EMAILID="$EMAILID" jq -n 'env.EMAILID'
The output is a JSON string:
"foo#bar.com"
I resolved this issue by escaping the inner double quotes
projectID=$(cat file.json | jq -r ".resource[] | select(.username==\"$EMAILID\") | .id")
Little unrelated but I will still put it here,
For other practical purposes shell variables can be used as -
value=10
jq '."key" = "'"$value"'"' file.json
Posting it here as it might help others. In string it might be necessary to pass the quotes to jq. To do the following with jq:
.items[] | select(.name=="string")
in bash you could do
EMAILID=$1
projectID=$(cat file.json | jq -r '.resource[] | select(.username=='\"$EMAILID\"') | .id')
essentially escaping the quotes and passing it on to jq
It's a quote issue, you need :
projectID=$(
cat file.json | jq -r ".resource[] | select(.username=='$EMAILID') | .id"
)
If you put single quotes to delimit the main string, the shell takes $EMAILID literally.
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". Use 'single quotes' for code or literal $'s: 'Costs $5 US', ssh host 'echo "$HOSTNAME"'. See
http://mywiki.wooledge.org/Quotes
http://mywiki.wooledge.org/Arguments
http://wiki.bash-hackers.org/syntax/words
Jq now have better way to access environment variables, you can use env.EMAILID:
projectID=$(cat file.json | jq -r ".resource[] | select(.username==env.EMAILID) | .id")
Another way to accomplish this is with the jq "--arg" flag.
Using the original example:
#!/bin/sh
#this works ***
projectID=$(cat file.json | jq -r '.resource[] |
select(.username=="myemail#hotmail.com") | .id')
echo "$projectID"
EMAILID=myemail#hotmail.com
# Use --arg to pass the variable to jq. This should work:
projectID=$(cat file.json | jq --arg EMAILID $EMAILID -r '.resource[]
| select(.username=="$EMAILID") | .id')
echo "$projectID"
See here, which is where I found this solution:
https://github.com/stedolan/jq/issues/626
I know is a bit later to reply, sorry. But that works for me.
export K8S_public_load_balancer_url="$(kubectl get services -n ${TENANT}-production -o wide | grep "ingress-nginx-internal$" | awk '{print $4}')"
And now I am able to fetch and pass the content of the variable to jq
export TF_VAR_public_load_balancer_url="$(aws elbv2 describe-load-balancers --region eu-west-1 | jq -r '.LoadBalancers[] | select (.DNSName == "'$K8S_public_load_balancer_url'") | .LoadBalancerArn')"
In my case I needed to use double quote and quote to access the variable value.
Cheers.
I also faced same issue of variable substitution with jq. I found that --arg is the option which must be used with square bracket [] otherwise it won't work.. I am giving you sample example below:
RUNNER_TOKEN=$(aws secretsmanager get-secret-value --secret-id $SECRET_ID | jq '.SecretString|fromjson' | jq --arg kt $SECRET_KEY -r '.[$kt]' | tr -d '"')
In case where we want to append some string to the variable value and we are using the escaped double quotes, for example appending .crt to a variable CERT_TYPE; the following should work:
$ CERT_TYPE=client.reader
$ cat certs.json | jq -r ".\"${CERT_TYPE}\".crt" #### This will *not* work #####
$ cat certs.json | jq -r ".\"${CERT_TYPE}.crt\""