I have a curl where I would like to extract two text to two different files, file1.crt and file2.key. currently configured this way by repeating curl.
curl -X GET -H "X-Vault-Token:{{ vault_token }}" https://fopp.com/v1/ACME/data/SSL/fopp.com | jq -r .data.data.crt > /files/nginx/ssl/file1.crt &&
curl -X GET -H "X-Vault-Token:{{ vault_token }}" https://fopp.com/v1/ACME/data/SSL/fopp.com | jq -r .data.data.key > /files/nginx/ssl/file2.key
I would like to know if with jq could handle to extract in just one command.
Not with only jq itself, but it's easy enough to combine with a bit of shell. If your values can't contain literal newlines, this can be as easy as:
curl -X GET -H "X-Vault-Token:{{ vault_token }}" https://fopp.com/v1/ACME/data/SSL/fopp.com \
| jq -r '.data.data.crt, .data.data.key' \
| { IFS= read -r crt && printf '%s\n' "$crt" > /files/nginx/ssl/file1.crt;
IFS= read -r key && printf '%s\n' "$key" > /files/nginx/ssl/file2.key; }
If the values can contain newlines, then you need to use a different separator. Consider:
curl -X GET -H "X-Vault-Token:{{ vault_token }}" https://fopp.com/v1/ACME/data/SSL/fopp.com \
| jq -j '.data.data.crt, "\u0000", .data.data.key, "\u0000"' \
| { IFS= read -r -d '' crt && printf '%s\n' "$crt" > /files/nginx/ssl/file1.crt;
IFS= read -r -d '' key && printf '%s\n' "$key" > /files/nginx/ssl/file2.key; }
Related
When I use
<some commands that output a json> | jq -r '.Credentials | .AccessKeyId, .SecretKey, .SessionToken'
I get the following output:
ABCDEF
123456
AAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBB
Three distinct lines with various keys.
I would like to export these outputs as exports:
export AWS_ACCESS_KEY_ID=<the first line of the output>
export AWS_SECRET_KEY=<the second line of the output>
export AWS_SESSION_TOKEN=<the third line of the output>
How do I do that (and still remain with oneliner)?
I tried doing the following:
<some commands that output a json> | jq -r '.Credentials | .AccessKeyId, .SecretKey, .SessionToken' | export AWS_ACCESS_KEY_ID=`awk 'NR==1'`
and it works but
<some commands that output a json> | jq -r '.Credentials | .AccessKeyId, .SecretKey, .SessionToken' | export AWS_ACCESS_KEY_ID=`awk 'NR==1'; export AWS_SECRET_KEY=`awk 'NR==2'`
hangs.
I'm using zsh.
An option not yet discussed by other answers is using the jq #sh filter to generate code that's directly safe to eval.
eval "$(printf '%s\n' "$json" | jq -r '
.Credentials | ("export AWS_ACCESS_KEY=\(.AccessKeyId | #sh)",
"export AWS_SECRET_KEY=\(.SecretKey | #sh)",
"export AWS_SESSION_TOKEN=\(.SessionToken | #sh)")')"
Note that the above could trivially be one line, and was broken up to generate three separate shell commands only for the sake of readability:
eval "$(printf '%s\n' "$json" | jq -r '.Credentials | "export AWS_ACCESS_KEY=\(.AccessKeyId | #sh) AWS_SECRET_KEY=\(.SecretKey | #sh) AWS_SESSION_TOKEN=\(.SessionToken | #sh)"')"
One advantage of this approach, which no other answers currently provide as-written, is that it will correctly handle keys or tokens that contain literal newlines.
As suggested by Charles Duffy, you can use something like this:
{ read -r AWS_ACCESS_KEY_ID && read -r AWS_SECRET_KEY && read -r AWS_SESSION_TOKEN; } << EOF
$(<some commands that output a json> | jq -r '.Credentials | .AccessKeyId, .SecretKey, .SessionToken')
EOF
export AWS_ACCESS_KEY_ID AWS_SECRET_KEY AWS_SESSION_TOKEN
Also, as suggested by Charles, you can use a here string, like this:
{ read -r AWS_ACCESS_KEY_ID && read -r AWS_SECRET_KEY && read -r AWS_SESSION_TOKEN; } <<<"$(some commands that output a json | jq -r '.Credentials | .AccessKeyId, .SecretKey, .SessionToken')"
export AWS_ACCESS_KEY_ID AWS_SECRET_KEY AWS_SESSION_TOKEN
And here is a proof of concept:
$ unset a b c
$ { read -r a && read -r b && read -r c; }<< EOF
$(cat t.txt)
EOF
$ echo $a $b $c
ABCDEF 123456 AAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBB
$ unset a b c
$ { read -r a && read -r b && read -r c; }<<<"$(cat t.txt)"
$ echo $a $b $c
ABCDEF 123456 AAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBBAAAAAABBBBB
I'd do it like this:
for i in AWS_ACCESS_KEY_ID AWS_SECRET_KEY AWS_SESSION_TOKEN; do
read "$i" &&
export "$i"
done \
< <(json commands |
jq -r '...')
The variables are only exported if something is successfully read. If you want them exported regardless (empty), just remove the "and" operator (&&).
Example here:
gitrepo=$(jq -r '.gitrepo' 0.json)
releasetag=$(curl --silent ""https://api.github.com/repos/\"$gitrepo\""/releases/latest" | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')
echo "$releasetag"
Used \" to escape characters.
0.json:
{
"type": "github-releases",
"gitrepo": "ipfs/go-ipfs"
}
How to put $gitrepo to work inside $releasetag?
Thanks in advance!
Bash variables expand inside quoted " strings.
gitrepo="$(jq -r '.gitrepo' 0.json)"
releasetag="$(
curl --silent "https://api.github.com/repos/$gitrepo/releases/latest" \
| grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/'
)"
echo "$releasetag"
Btw, as you are using jq to extract .gitrepo from 0.json, you could also use it in the exact same way to extract .tag_name from curl's output (instead of using grep and sed) like so:
gitrepo="$(jq -r '.gitrepo' 0.json)"
releasetag="$(
curl --silent "https://api.github.com/repos/$gitrepo/releases/latest" \
| jq -r '.tag_name'
)"
echo "$releasetag"
And to simplify it even further (depending on your use case), just write:
curl --silent "https://api.github.com/repos/$(jq -r '.gitrepo' 0.json)/releases/latest" \
| jq -r '.tag_name'
I'd like to pick two values from the text file and paste it to another file after certain strings:
#!/bin/bash
IAM_ROLE=$(curl --silent http://169.254.169.254/latest/meta-data/iam/security-credentials)
curl --silent http://169.254.169.254/latest/meta-data/iam/security-credentials/$IAM_ROLE | jq -r '.AccessKeyId, .SecretAccessKey' > /tmp/aws_credentials
curl --silent http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region > /tmp/aws_region
{ read -r val1
read -r val2
sed -i 's! access_key= .* *$! access_key= $val1 !; s! secret_key= .* *$! secret_key= $val2 !;' /etc/trafficserver/s3_auth_v4.config
} < /tmp/aws_credentials
/etc/trafficserver/s3_auth_v4.config looks like:
access_key=
secret_key=
version=4
v4-region-map=region_map.config
However sed part of the script is doing nothing.
So, I'm trying to export an environment variable that comes from an api that returns json values. Would like to use jq to just do a one liner, but if the values have spaces I cannot get it working
Trying without surrounding the value in quotes
/app/src $ $(echo '{"params":[{ "Name":"KEY","Value":"value with space"}]}' | jq
-r '.params[] | "export " + .Name + "=" + .Value')
/app/src $ printenv KEY
value
/app/src $
Next, I try wrapping the value in quotes
/app/src $ $(echo '{"params":[{ "Name":"KEY","Value":"value with space"}]}' | jq
-r '.params[] | "export " + .Name + "=\"" + .Value + "\""')
sh: export: space": bad variable name
/app/src $
For all of the below, I'm assuming that:
json='{"params":[{ "Name":"KEY","Value":"value with space"}]}'
It can be done, but ONLY IF YOU TRUST YOUR INPUT.
A solution that uses eval might look like:
eval "$(jq -r '.params[] | "export \(.Name | #sh)=\(.Value | #sh)"' <<<"$json")"
The #sh builtin in jq escapes content to be eval-safe in bash, and the eval invocation then ensures that the content goes through all parsing stages (so literal quotes in the data emitted by jq become syntactic).
However, all solutions that allow arbitrary shell variables to be assigned have innate security problems, as the ability to set variables like PATH, LD_LIBRARY_PATH, LD_PRELOAD and the like can be leveraged into arbitrary code execution.
Better form is to generate a NUL-delimited key/value list...
build_kv_nsv() {
jq -j '.params[] |
((.Name | gsub("\u0000"; "")),
"\u0000",
(.Value | gsub("\u0000"; "")),
"\u0000")'
}
...and either populate an associative array...
declare -A content_received=( )
while IFS= read -r -d '' name && IFS= read -r -d '' value; do
content_received[$name]=$value
done < <(build_kv_nsv <<<"$json")
# print the value of the populated associative array
declare -p content_received
...or to use a namespace that's prefixed to guarantee safety.
while IFS= read -r -d '' name && IFS= read -r -d '' value; do
printf -v "received_$name" %s "$value" && export "received_$name"
done < <(build_kv_nsv <<<"$json")
# print names and values of our variables that start with received_
declare -p "${!received_#}" >&2
If the values are known not to contain (raw) newlines, and if you have access to mapfile, it may be worthwhile considering using it, e.g.
$ json='{"params":[{ "Name":"KEY","Value":"value with space"}]}'
$ mapfile -t KEY < <( jq -r '.params[] | .Value' <<< "$json" )
$ echo N=${#KEY[#]}
N=1
If the values might contain (raw) newlines, then you'd need a version of mapfile with the -d option, which could be used as illustrated below:
$ json='{"params":[{ "Name":"KEY1","Value":"value with space"}, { "Name":"KEY2","Value":"value with \n newline"}]}'
$ mapfile -d $'\0' KEY < <( jq -r -j '.params[] | .Value + "\u0000"' <<< "$json" )
$ echo N=${#KEY[#]}
N=2
I need to set or access multiple cURL variables so I can access them later in a script. For example:
curl -s --write-out "%{http_code} | %{local_ip} | %{time_total}" "http://endpoint.com/payload"
Now how can I access http_code or local_ip to do things like add them to an bash array, etc? Is the only option to grep them out of the response?
You can pipe your curl command to a read command :
curl -s --write-out "write-out: %{http_code} | %{local_ip} | %{time_total}\n" "http://yahoo.com" | \
sed -n '/^write-out:/ s///p' | \
while IFS='|' read http_code local_ip time_total;
do
printf "http_code: %s\nlocal_ip: %s\ntotal_time: %s\n" $http_code $local_ip $time_total;
# or in an array
curlvars=($http_code $local_ip $time_total)
for data in "${curlvars[#]}"
do
printf "%s | " $data
done
done
I added a \n to the write-out string to allow process it as a line.
The sed command extract the write-out line from the curl output.
In the read command you can define a separator and assign all parsed strings to vars.