How to interpolate several values contained in a variable into a json string - bash

I have a parsed variable obtained after parsing some text:
parsed=$(echo "PA-232 message1 GX-1234 message2 PER-10 message3" | grep -Eo '[A-Z]+-[0-9]+')
parsed contains a bunch of ids:
echo $parsed
PA-232
GX-1234
PER-10
The next thing I have to do in my script is generate a json text and invoke an API with it:
The json text should be
"{\"tasks\": [{\"taskId\": \"PA-232\"}, {\"taskId\": \"GX-1234\"}, {\"taskId\": \"PER-10\"}], \"projectId\": \"$CI_PROJECT_ID\" }"
Notice CI_PROJECT_ID is an envvar that I also have to send, thats why I needed to use double quotes and escape them.
And it would be called with curl:
curl -X POST -H 'Content-Type:application/json' -k -u $CLIENT_ID:$CLIENT_SECRET 'https://somewhere.com/api/tasks' -d "{\"tasks\": [{\"taskId\": \"PA-232\"}, {\"taskId\": \"GX-1234\"}, {\"taskId\": \"PER-10\"}], \"projectId\": \"$CI_PROJECT_ID\"}"
The question is how can I generate a json string like the one shown above from the parsed variable and the additional envvar?

How about doing it with jq?
CI_PROJECT_ID='I want this " to be escaped automatically'
echo 'PA-232 message1 GX-1234 message2 PER-10 message3' |
jq -R --arg ciProjectId "$CI_PROJECT_ID" '
{
tasks: [
capture( "(?<taskId>[[:upper:]]+-[[:digit:]]+)"; "g" )
],
projectId: $ciProjectId
}
'
{
"tasks": [
{
"taskiD": "PA-232"
},
{
"taskiD": "GX-1234"
},
{
"taskiD": "PER-10"
}
],
"projectId": "I want this \" to be escaped automatically"
}
note: you can use jq -c ... for outputting a compact JSON
And here's a solution without jq that doesn't escape the characters in the strings so it might generate invalid JSON:
CI_PROJECT_ID='no escaping needed'
tasks_jsonArr=$(
echo "PA-232 message1 GX-1234 message2 PER-10 message3" |
grep -Eo '[A-Z]+-[0-9]+' |
sed 's/.*/{ "taskiD": "&" }/' |
paste -sd ',' |
sed 's/.*/[ & ]/'
)
curl -k 'https://somewhere.com/api/tasks' \
-X POST \
-H 'Content-Type:application/json' \
-u "$CLIENT_ID:$CLIENT_SECRET" \
-d "{\"tasks\": $tasks_jsonArr, \"projectId\": \"$CI_PROJECT_ID\"}"
N.B. For JSON-escaping strings with standard tools, take a look at function json_stringify in awk

Related

jq format when running from a bash script with variable expansion

I've got a jq command that works when running directly from the shell or from within a shell script, but when I try to add variable expansion, I get jq errors for unexpected format or invalid characters. My goal is to have a quick and easy way to update some json configuration.
Here's a simplified example.
The format of the json I'm modifying:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
}
The jq without variable expansion:
cat myfile.json | jq '.pets."some-new-pet" += {PetInfo: {name: "my-brand-new-pet"}, toys: ["toy1", "toy2", "toy3"]}}'
The above runs fine, and adds the new pets.some-new-pet entry to my json.
Below is what I'm trying to do with variable expansion that fails.
jq_args = "'.pets.\"${PET}\" += {PetInfo: {name: \"${NAME}\"}, toys: [\"${toy1}-postfix\", \"${toy2}-postfix\", \"${toy3}-postfix\"]}}'"
cat myfile.json | jq $jq_args
The error message I get with the above:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1: '.pets."some-new-pet"
My file is formatted as utf-8 and uses LF line endings.
I do not recommend constructing a jq filter using variable expansion or printf. It will work for simple cases but will fail if the string contains double quotes, backslashes or control-codes, as they have special meanings inside a JSON string. As an alternative to using printf, jq has a way to pass in variables directly via the command-line, avoiding all these issues.
pet='some-second-pet'
name='my-even-newer'
toy1=toy1
toy2=toy2
toy3=toy3
jq \
--arg pet "$pet" \
--arg name "$name" \
--arg toy1 "$toy1" \
--arg toy2 "$toy2" \
--arg toy3 "$toy3" \
'.pets[$pet] += {
PetInfo: {name: $name},
toys: ["\($toy1)-postfix", "\($toy2)-postfix", "\($toy3)-postfix"]
}' \
myfile.json
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
},
"some-second-pet": {
"PetInfo": {
"name": "my-even-newer-pet"
},
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
It would be cleaner and less error prone to format the string using printf
PET='dog'
NAME='sam'
toy1="t1"
toy2="t2"
toy3="t3"
jq_args=$(printf '.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s-postfix", "%s-postfix", "%s-postfix"]}}' "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}")
echo "$jq_args"
Result:
.pets."dog" += {PetInfo: {name: "sam"}, toys: ["t1-postfix", "t2-postfix", "t3-postfix"]}
Additionally, redundant quoting could be avoided by quoting the arg on this command
cat myfile.json | jq "$jq_args"
Fix your jq code by removing extra } at end
Fix bash jq call:
Add cotes "..." around your $jq_args
so don't use singles '...' in your jq_args definition
Use printf with -v option to define jq_args:
printf -v jq_args "...format..." value1 value2 ...
So your code became:
PET="some-new-pet"
NAME="my-brand-new-pet"
toy1="toy1"
toy2="toy2"
toy3="toy3"
format='.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s", "%s", "%s"]}'
printf -v jq_args "${format}" "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}"
cat myfile.json | jq "$jq_args"
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet"
},
"toys": [
"toy1",
"toy2",
"toy3"
]
}
}
}
Notes:
When you define your format, you put it into simple cotes '...'. It's really better to format JSON (or XML) without back-slashes (\\) before each double cotes (")
Use printf -v variable_name. It's more readable than var_name=$(printf ...)
By constructing the jq filter ("code") using outer bash variables ("data") you may run into escaping issues, which could eventually break or even divert your filter. (see https://en.wikipedia.org/wiki/Code_injection)
Instead, use mechanisms by jq to introduce external data through variables (parameter --arg):
jq --arg pet "${PET}" \
--arg name "${NAME}" \
--arg toy1 "${toy1}-postfix" \
--arg toy2 "${toy2}-postfix" \
--arg toy3 "${toy3}-postfix" \
'
.pets[$pet] += {PetInfo: {$name, toys: [$toy1,$toy2,$toy3]}}
' myfile.json
If you have an unknown number of variables to include, check out jq's --args parameter (note the additional s)

argument list too long curl

Trying to solve "argument list too long"
I have been searching for a solution and found the closest one to my issue
curl: argument list too long
however the response is not clear as I am still having the issue "argument list too long"
curl -X POST -d #data.txt \
https://Path/to/attachments \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY' \
-d '{
"data": {
"type": "attachments",
"attributes": {
"attachment": {
"content": "'$(cat data.txt | base64 --wrap=0)'",
"file_name": "'"$FileName"'"
}
}
}
}'
thank you
Use jq to format your base64 encoded data string into a proper JSON string, and then pass the JSON data as standard input to the curl command.
#!/usr/bin/env sh
attached_file='img.png'
# Pipe the base64 encoded content of attached_file
base64 --wrap=0 "$attached_file" |
# into jq to make it a proper JSON string within the
# JSON data structure
jq --slurp --raw-input --arg FileName "$attached_file" \
'{
"type": "attachments",
"attributes": {
"attachment": {
"content": .,
"file_name": $FileName
}
}
}
' |
# Get the resultant JSON piped into curl
# that will read the data from the standard input
# using -d #-
curl -X POST -d #- \
'https://Path/to/attachments' \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY'
Per the linked answer
you are trying to pass the entirety of the base64'd content on the command line
This is a limitation of the shell, not curl. That is, the shell is responding with error argument list too long. The program curl is never even started.
The recommendation is
curl has the ability to load in data to POST from a file
Write the json data to some file /tmp/data.json using piping.(the commands will use piping | and file redirection > >> which can handle arbitrarily large amounts of data. Whereas, you cannot place arbitrarily large amounts of data into a single command, there is a limit).
echo -n '
{
"data": {
"type": "attachments",
"attributes": {
"attachment": {
"content": "' > /tmp/data.json
cat data.txt | base64 --wrap=0 >> /tmp/data.json
echo -n '",
"file_name": "'"$FileName"'"
}
}
}
}' >> /tmp/data.json
Pass that file path /tmp/data.json to the curl command using # so curl knows it's a file path.
curl -X POST -d #/tmp/data.json \
"https://Path/to/attachments" \
-H 'content-type: application/vnd.api+json' \
-H 'x-api-key: KEY'

how to build json without escaping new line?

I run jq from bash and all my new lines are escaped
release_message="\`\`\`a\na\n\`\`\`"
query=$(jq -n \
--arg message $release_message \
"{text:\$message}"
);
echo "query $query"
result
query {
"text": "```a\\na\\n```"
}
How to prevent extra escape from jq?
You can either encode your input in JSON yourself or let jq do it
option 1 : encode yourself
release_message='"```a\na\n```"'
jq -n --argjson message "$release_message" '{text:$message}'
# or :
# echo "$release_message" | jq '{text:.}'
Have bash produce a valid JSON string (note : quotes-enclosed), pass through the standard input or with --argjson.
option 2 : let jq encode the string
release_message='```a
a
```'
jq --arg message "$release_message" '{text:$message}'
# or :
# echo "$release_message" | jq -R --slurp '{text:.}'
Have bash produce the literal string, pass with --arg or specify --raw-input/-R to have the input encoded in JSON, plus --slurp so that the multiple lines are considered as a single string.
Since you're using bash, it's generally best to use single quotes unless you want string interpolation. Consider, for example:
release_message='```a\na\n```'
query=$(jq -n \
--arg message "$release_message" \
'{text: $message }'
);
echo "query $query"
You might want to consider using $'....':
release_message=$'```a\na\n```'
Using gsub
Depending on what your actual goals are, you might want to use gsub, e.g.
release_message='```a\na\n```'
query=$(jq -n \
--arg message "$release_message" \
'{text: ($message | gsub("\\n";"\n")) }'
);
echo "query $query"
produces:
query {
"text": "```a\\na\\n```"
}

How to fetch next URL from JSON using Bash

I have created a simple script to store response from a third party API
The request is like this..
https://externalservice.io/orders?key=password&records=50&offset=0
The response is as follows:
{
"results": [
{
"engagement": {
"id": 29090716,
"portalId": 62515,
"active": true,
"createdAt": 1444223400781,
"lastUpdated": 1444223400781,
"createdBy": 215482,
"modifiedBy": 215482,
"ownerId": 70,
"type": "NOTE",
"timestamp": 1444223400781
},
},
],
"hasMore": true,
"offset": 4623406
}
If there is a hasMore attribute, I need to read the offset value to get the next set of records.
Right now I've created a script that simply loops over the estimated number of records (I believed there is) and thought incrementing the offset would work but this is not the case as the offset is not incremental.
#!/bin/bash
for i in {1..100}; do
curl -s "https://externalservice.io/orders?key=password&records=50&offset=$i" >>outfile.txt 2>&1
done
Can someone explain how I can read continue the script reading the offset value until hasMore=false?
You can read a value from a json using the jq utility:
$ jq -r ".hasMore" outfile
true
Here is what you could use:
more="true"
offset=0
while [ $more = "true" ]; do
echo $offset
response=$(curl -s "https://example.com/orders?offset=$offset")
more=$(echo $response | tr '\r\n' ' ' | jq -r ".hasMore")
offset=$(echo $response | tr '\r\n' ' ' | jq -r ".offset")
done
You can use jq utility to extract specific attribute from you json response:
$ jq -r ".hasMore" outfile
true
jq expects perfectly valid json input, otherwise it will print error.
Most common mistake is to echo json stored in variable. A mistake, because echo will interpret all escaped newlines in your json and will send unescaped newlines within values causing jq to throw parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line message.
To avoid modification to json (and avoiding an error) we need to use printf instead of echo.
Here is the solution without breaking json content:
more="true"
offset=0
while [ $more = "true" ]; do
echo $offset
response=$(curl -s "https://example.com/orders?offset=$offset")
more=$(printf "%s\n" "$response" | jq -r ".hasMore")
offset=$(printf "%s\n" "$response" | jq -r ".offset")
done

ShellScript to send Mattermost notification is not working

I want to send a Message into a Mattermost channel with the help of a ShellScript/WebHook/cURL. The following code is the function to send the Message.
function matterSend() {
ENDPOINT=https://url.to.Mattermost/WebhookID
USERNAME="${USER}"
PAYLOAD=$(cat <<'EOF'
'payload={
"username" : "${USERNAME}",
"channel" : "TestChannel",
"text" : "#### Test to \n
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
}'
EOF
)
echo "CURL: curl -i -X POST -d ${PAYLOAD} ${ENDPOINT}"
curl -i -X POST -d "${PAYLOAD}" "${ENDPOINT}"
}
As you can see, when I ECHO the command I get:
curl -i -X POST -d 'payload={
"username" : "TestUser",
"channel" : "TestChannel",
"text" : "#### Test to \n
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
}' https://url.to.Mattermost/WebhookID
If I paste that code directly into the terminal and execute it, it works. But when I run the script with the help of a Jenkins-Job I get the Error:
Unable to parse incoming data","message":"Unable to parse incoming
data".
Why is it not working?
Without knowledge of the API you are connecting to, I would guess that you need
# Drop function keyword, indent body
matterSend() {
# Lowercase variable names; declare them local
local endpoint=https://url.to.Mattermost/WebhookID
local username=$USER
# Pro tip: don't use a variable for the payload if it's effectively static
payload=$(cat <<-__EOF
payload={
"username" : "$username",
"channel" : "TestChannel",
"text" : "#### Test to \\n| TestR | TestS | New Mode |\\n|:-----------|:-----------:|-----------------------------------------------:|\\n| ${2} | ${3} | ${1} :white_check_mark: |\\n"
}
__EOF
)
echo "CURL: curl -i -X POST -d $payload $endpoint"
curl -i -X POST -d "$payload" "$endpoint"
}
Replacing the newlines inside the "text" element with \n (and doubling the backslash because the here document is now being interpreted by the shell when it's assigned) is mildly speculative; perhaps remove the remaining newlines, too. The real beef is removing the misplaced literal single quotes around the payload.
Maybe also explore printf for formatting fixed-width tabular text.
The here document's <<-__EOF uses the unquoted separator __EOF and the dash before it says to remove any tabs from the beginning of each line. Needless to say, the indentation on those lines consists of a literal tab character.
Generating JSON (or XML, or other structured formats) via string concatenation leads to pain and suffering. Instead, use a tool that actually understands the format.
Using a compliant generator such as jq means you no longer need to be responsible for putting \ns in the data (as multi-character strings), changing "s in text to \", or any of the other otherwise-necessary munging.
matterSend() {
# Lowercase variable names; declare them local
local endpoint=https://url.to.Mattermost/WebhookID
local username=$USER
local text="#### Test to
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
payload=$(jq --arg username "$username" \
--arg channel "TestChannel" \
--arg text "$text" \
'{"username": $username, "channel": $channel, "text": $text}')
# Advice: use "set -x" if you want to trace the commands your script would run.
# ...or at least printf %q, as below; avoids misleading output from echo.
# printf '%q ' curl -i -X POST -d "$payload" "$endpoint" >&2; echo >&2
curl -i -X POST -d "$payload" "$endpoint"
}

Resources