I want to use this command into a bash script where each time I will have a different array input containing the parameters?
knowing that I have an array (as input from the user) where each column contains "parameteri=valuei".
I want to get rid of the hardcoded aspect in introducing the name and the value of each parameter.
For instance, with this input:
"id=123,verbosity=high"
I will eventually get this final instruction:
curl -X POST JENKINS_URL/job/JOB_NAME/build \
--user USER:TOKEN \
--data-urlencode json='{"parameter": [{"name":"id", "value":"123"}, {"name":"verbosity", "value":"high"}]}'
What is a clean way to do so?
You can make it the sexy way, building the jsonParameters from specified key=value parameters:
#!/bin/bash
jsonParameters=""
while IFS=',' read -r -a parameterEntries; do
for parameterEntry in "${parameterEntries[#]}"; do
IFS='=' read -r key value <<< "$parameterEntry"
[ ! -z "$jsonParameters" ] && jsonParameters="$jsonParameters,"
jsonParameters="$jsonParameters {\"name\":\"$key\", \"value\": \"$value\"}"
done
done <<< "$#"
Explanations:
the first loop will create the array named parameterEntries, with all your specified parameters, each element will contain key=value
then, the second loop, which iterates on each element of this array, will extract key, and value of it
eventually, it is only syntax writting to get the JSON output you want
the [ ! -z "$jsonParameters" ] && jsonParameters="$jsonParameters," is just here to add a separating coma, only if there is more than one element
Then you simply have to use the $jsonParameters where you want:
curl -X POST JENKINS_URL/job/JOB_NAME/build \
--user USER:TOKEN \
--data-urlencode json="{\"parameter\": [$jsonParameters]}"
Related
I am using the bash code below to store the result of a curl command in a text file.
cat /c/customer_files/Bain/artifacts1.txt
sample=$(curl -X GET --header 'Accept: application/json' --header 'Authorization: Bearer <my_token>' '<api_>' | jq -r '.items[] | .id')
echo "$sample" >> /c/my_files/artifacts1.txt
This generates a text file with content below:
606b69cff140fe0d98e78d2a
60a40910c403d464225343b5
607f1e14d514043adcf4a0f6
60c36c380093aa519b816554
Now, I want to iterate through this file line by line. I am using the code below to do that.
while read -r line; do
#reading each line
echo "Line No. $n : $line";
n=$((n+1))
This is producing correct result as expected.
Line No. 1 : 606b69cff140fe0d98e78d2a
Line No. 2 : 60a40910c403d464225343b5
Line No. 3 : 607f1e14d514043adcf4a0f6
Line No. 4 : 60c36c380093aa519b816554
I want to pass variable $line to CURL command via json body below.
input_json="{"executable": "<some ID>", "keepTargetResources": true,"keepTargetRunProfiles": true,"advanced": {"artifactId": "$line","artifactType": "ACTION"}}"
However, this produces a result below:
,artifactType: ACTION}}d6a50bb75bbe81, keepTargetResources: true,keepTargetRunProfiles:
true,advanced: {artifactId: 606b69cff140fe0d98e78d2a
,artifactType: ACTION}}d6a50bb75bbe81, keepTargetResources: true,keepTargetRunProfiles:
true,advanced: {artifactId: 60a40910c403d464225343b5
,artifactType: ACTION}}d6a50bb75bbe81, keepTargetResources: true,keepTargetRunProfiles:
true,advanced: {artifactId: 607f1e14d514043adcf4a0f6
{executable: 60ca3bf02ed6a50bb75bbe81, keepTargetResources: true,keepTargetRunProfiles:
true,advanced: {artifactId: 60c36c380093aa519b816554,artifactType: ACTION}}
It creates an output in the desired format only for the last record:
{executable: 60ca3bf02ed6a50bb75bbe81, keepTargetResources:
true,keepTargetRunProfiles:true,advanced: {artifactId:
60c36c380093aa519b816554,artifactType: ACTION}}
For 1st 3 records, it looks like it is overwriting content at the start of the line.
What am I doing wrong? Please advice.
Thanks in Advance.
JSON needs to have its variables and values surrounded with ". Use \ to prevent the shell to interpret the ".
The extra carriage return is a mystery...
Maybe an extra clean up of line could make it
Prefer printf to echo and use "${line}" for more safety.
Give this a try:
artifact_id="$(printf "%s" "${line}" | sed 's/^\(.*[^[:blank:]]\)[[:blank:]]*$/\1/g')"
input_json="{\"executable\": \"<some ID>\", \"keepTargetResources\": true,\"keepTargetRunProfiles\": true,\"advanced\": {\"artifactId\": \"${artifact_id}\",\"artifactType\": \"ACTION\"}}"
I would like to get the values from Json file. Which is working.
JsonFileToTest:
{
"permissions": [
{
"emailid": "test1#test.com",
"rights": "read"
},
{
"emailid": "test2#test.com",
"rights": "read"
}
]
}
readPermissions=($(jq -r '.permissions' JsonFileToTest))
# The command below works perfectly, But when I Put it in a loop, It does not.
#echo ${readPermissions[#]} | jq 'values[].emailid'
for vals in ${readPermissions[#]}
do
# I would like o extract the email id of the user. The loop is not working atm.
echo ${vals[#]} | jq 'values[].emailid'
done
what am I missing here?
thanks
If you really want to do it this way, that might look like:
readarray -t permissions < <(jq -c '.permissions[]' JsonFileToTest)
for permissionSet in "${permissions[#]}"; do
jq -r '.emailid' <<<"$permissionSet"
done
Note that we're telling jq to print one line per item (with -c), and using readarray -t to read each line into an array element (unlike the array=( $(...command...) ) antipattern, which splits not just on newlines but on other whitespace as well, and expands globs in the process).
But there's no reason whatsoever to do any of that. You'll get the exact same result simply running:
jq -r '.permissions[].emailid' JsonFileToTest
In my bash script, when I run the following jq against my curl result:
curl -u someKey:someSecret someURL 2>/dev/null | jq -r '.schema' | jq -r -c '.fields'
I get back a JSON array as follows:
[{"name":"id","type":"int","doc":"Documentation for the id field."},{"name":"test_string","type":"string","doc":"Documentation for the test_string field"}]
My goal is to do a call with jq applied to return the following (given the example above):
{"id":1234567890,"test_string":"xxxxxxxxxx"}
NB: I am trying to automatically generate templated values that match the "schema" JSON shown above.
So just to clarify, that is:
all array objects (there could be more than 2 shown above) returned in a single comma-delimited row
doc fields are ignored
the values for "name" (including their surrounding double-quotes) are concatenated with either:
:1234567890 ...when the "type" for that object is "int"
":xxxxxxxxxx" ...when the "type" for that object is "string"
NB: these will be the only types we ever get for now
Can someone show me how I can expand upon my initial jq to return this?
NB: I tried working down the following path but am failing beyond this...
curl -u someKey:someSecret someURL 2>/dev/null | jq -r '.schema' | jq -r -c '.fields' | "\(.name):xxxxxxxxxxx"'
If it's not possible in pure JQ (my preference) I'm also happy for a solution that mixes in a bit of sed/awk magic :)
Cheers,
Stan
Given the JSON shown, you could add the following to your pipeline:
jq -c 'map({(.name): (if .type == "int" then 1234567890 else "xxxxxxxxxx" end)})|add'
With that JSON, the output would be:
{"id":1234567890,"test_string":"xxxxxxxxxx"}
However, it would be far better if you combined the three calls to jq into one.
Long story short, I will be parsing yaml files in a directory with bash using yq. My yaml files could look like this:
CLIENT_FIRST_NAME: bob
CLIENT_LAST_NAME: smith
Or
CLIENT_FIRST_NAME: bob
CLIENT_LAST_NAME: smith
CLIENT_MIDDLE_NAME: michael
So I am looping through each file with a do loop and setting the variables to values
For example:
for f in $FILES
do
FIRSTNAME=$(yq r $f CLIENT_FIRST_NAME)
LASTNAME=$(yq r $f CLIENT_LAST_NAME)
add client --firstname=${FIRSTNAME} --lastname=${LASTNAME}
done
But sometimes I will have that middle name and I would need to include that:
add client --firstname=${FIRSTNAME} --lastname=${LASTNAME} --middlename=${MIDDLENAME}
The order doesn't matter, I just need to be able to account for additional fields that may show up in the yaml that need to be added to the 'add client' command. EVERY line in the yaml will be added to the command. Every key added will be a viable parameter for the 'add client' command. I don't have to worry about whether or not a key in the yaml is a valid parameter. They WILL be.
Curious on the best approach to the unknown here. Thanks!
I'm assuming yq returns nothing if it doesn't find a key.
I might make the entire flag based on whether yq returns something, like
for f in "${FILES[#]}"
do
FIRSTNAME=$(yq r "$f" CLIENT_FIRST_NAME)
MIDDLENAME=$(yq r "$f" CLIENT_MIDDLE_NAME)
LASTNAME=$(yq r "$f" CLIENT_LAST_NAME)
[[ -n $MIDDLENAME ]] && MIDDLENAME="--middlename=${MIDDLENAME}"
add client --firstname="${FIRSTNAME}" --lastname="${LASTNAME}" "${MIDDLENAME}"
done
This code would be far more efficient if you only ran yq once per input file, not once per data item per input file. Consider:
for f in *.yml; do
{ read -r firstname; read -r middlename; read -r lastname; } < <(
yq -r '(.CLIENT_FIRST_NAME, .CLIENT_MIDDLE_NAME // "", .CLIENT_LAST_NAME)' "$f"
)
add client \
--firstname="$firstname" \
${middlename:+--middlename="$middlename"} \
--lastname="$lastname"
done
Some notes to use in reading this:
Each read command in bash reads one line, when -d is not used to modify this.
The above yq command outputs one line per data item.
Using // "" causes the empty string, instead of null, to be used when no CLIENT_MIDDLE_NAME is found.
${foo:+...words here...} expands to ...words here... if-and-only-if foo is set to a non-empty value.
So, I am getting a response from an API that I am calling in a shell script in the following form
[{"id":100000004,"name":"Customs Clearance Requested"},{"id":100000005,"name":"Customs Cleared"},{"id":100000006,"name":"Cargo Loaded to Vessel"}]
I want to create a map out of it that will help me lookup the id's from a name and use it in the shell script. So something like map["Customs Clearance Requested"] would give me 100000004 which I can use further. Can this be done using jq? I am pretty new to shell scripting and jq and got stuck with above thing
json='[{"id":100000004,"name":"Customs Clearance Requested"},{"id":100000005,"name":"Customs Cleared"},{"id":100000006,"name":"Cargo Loaded to Vessel"}]'
declare -A map
while IFS= read -r -d '' name && IFS= read -r -d '' value; do
map[$name]=$value
done < <(jq -j '.[] | "\(.name)\u0000\(.id)\u0000"' <<<"$json")
declare -p map # demo purposes: print the map we created as output
...emits as output:
declare -A map=(["Cargo Loaded to Vessel"]="100000006" ["Customs Clearance Requested"]="100000004" ["Customs Cleared"]="100000005" )
...which you can query exactly as requested:
$ echo "${map['Cargo Loaded to Vessel']}"
100000006
You could use the select function, e.g.:
data='[{"id":100000004,"name":"Customs Clearance Requested"},{"id":100000005,"name":"Customs Cleared"},{"id":100000006,"name":"Cargo Loaded to Vessel"}]'
jq 'map(select(.["name"] == "Customs Clearance Requested"))' <<< $data
It will get all elements which name equals "Customs Clearance Requested", e.g.:
[
{
"id": 100000004,
"name": "Customs Clearance Requested"
}
]
If you want to get the id field:
jq 'map(select(.["name"] == "Customs Clearance Requested")["id"])' <<< $data
This will output:
[
100000004
]
Please note that it will return an array and not a single element because the search does not know how many results will be found.
If you want to generalize this in a shell function, you could write:
function get_id_from_name
{
# $1=name to search for
local filter=$(printf 'map(select(.["name"] == "%s")["id"])' "$1")
jq "$filter"
}
Then call it like that:
get_id_from_name "Customs Clearance Requested" <<< $data
If your data is stored in a file, you could call it this way:
get_id_from_name "Customs Clearance Requested" < /path/to/file.json
The following is very similar to #CharlesDuffy's excellent answer but does not assume that the .name and .id values are NUL-free (i.e., do not have any "\u0000" characters):
declare -A map
while read -r name
do
name=$(sed -e 's/^"//' -e 's/"$//' <<< "$name")
read -r id
map[$name]="$id"
done < <(echo "$json" | jq -c '.[]|.name,.id')
The point is that the -j option is like -r (i.e., produces "raw output"), whereas the -c option produces JSON.
This means that if you don't want the .id values as JSON strings, then the above won't be a solution; also, if the .name values contain double-quotes, then you might want to deal with the occurrences of \".