how to build json without escaping new line? - bash

I run jq from bash and all my new lines are escaped
release_message="\`\`\`a\na\n\`\`\`"
query=$(jq -n \
--arg message $release_message \
"{text:\$message}"
);
echo "query $query"
result
query {
"text": "```a\\na\\n```"
}
How to prevent extra escape from jq?

You can either encode your input in JSON yourself or let jq do it
option 1 : encode yourself
release_message='"```a\na\n```"'
jq -n --argjson message "$release_message" '{text:$message}'
# or :
# echo "$release_message" | jq '{text:.}'
Have bash produce a valid JSON string (note : quotes-enclosed), pass through the standard input or with --argjson.
option 2 : let jq encode the string
release_message='```a
a
```'
jq --arg message "$release_message" '{text:$message}'
# or :
# echo "$release_message" | jq -R --slurp '{text:.}'
Have bash produce the literal string, pass with --arg or specify --raw-input/-R to have the input encoded in JSON, plus --slurp so that the multiple lines are considered as a single string.

Since you're using bash, it's generally best to use single quotes unless you want string interpolation. Consider, for example:
release_message='```a\na\n```'
query=$(jq -n \
--arg message "$release_message" \
'{text: $message }'
);
echo "query $query"
You might want to consider using $'....':
release_message=$'```a\na\n```'
Using gsub
Depending on what your actual goals are, you might want to use gsub, e.g.
release_message='```a\na\n```'
query=$(jq -n \
--arg message "$release_message" \
'{text: ($message | gsub("\\n";"\n")) }'
);
echo "query $query"
produces:
query {
"text": "```a\\na\\n```"
}

Related

How to interpolate several values contained in a variable into a json string

I have a parsed variable obtained after parsing some text:
parsed=$(echo "PA-232 message1 GX-1234 message2 PER-10 message3" | grep -Eo '[A-Z]+-[0-9]+')
parsed contains a bunch of ids:
echo $parsed
PA-232
GX-1234
PER-10
The next thing I have to do in my script is generate a json text and invoke an API with it:
The json text should be
"{\"tasks\": [{\"taskId\": \"PA-232\"}, {\"taskId\": \"GX-1234\"}, {\"taskId\": \"PER-10\"}], \"projectId\": \"$CI_PROJECT_ID\" }"
Notice CI_PROJECT_ID is an envvar that I also have to send, thats why I needed to use double quotes and escape them.
And it would be called with curl:
curl -X POST -H 'Content-Type:application/json' -k -u $CLIENT_ID:$CLIENT_SECRET 'https://somewhere.com/api/tasks' -d "{\"tasks\": [{\"taskId\": \"PA-232\"}, {\"taskId\": \"GX-1234\"}, {\"taskId\": \"PER-10\"}], \"projectId\": \"$CI_PROJECT_ID\"}"
The question is how can I generate a json string like the one shown above from the parsed variable and the additional envvar?
How about doing it with jq?
CI_PROJECT_ID='I want this " to be escaped automatically'
echo 'PA-232 message1 GX-1234 message2 PER-10 message3' |
jq -R --arg ciProjectId "$CI_PROJECT_ID" '
{
tasks: [
capture( "(?<taskId>[[:upper:]]+-[[:digit:]]+)"; "g" )
],
projectId: $ciProjectId
}
'
{
"tasks": [
{
"taskiD": "PA-232"
},
{
"taskiD": "GX-1234"
},
{
"taskiD": "PER-10"
}
],
"projectId": "I want this \" to be escaped automatically"
}
note: you can use jq -c ... for outputting a compact JSON
And here's a solution without jq that doesn't escape the characters in the strings so it might generate invalid JSON:
CI_PROJECT_ID='no escaping needed'
tasks_jsonArr=$(
echo "PA-232 message1 GX-1234 message2 PER-10 message3" |
grep -Eo '[A-Z]+-[0-9]+' |
sed 's/.*/{ "taskiD": "&" }/' |
paste -sd ',' |
sed 's/.*/[ & ]/'
)
curl -k 'https://somewhere.com/api/tasks' \
-X POST \
-H 'Content-Type:application/json' \
-u "$CLIENT_ID:$CLIENT_SECRET" \
-d "{\"tasks\": $tasks_jsonArr, \"projectId\": \"$CI_PROJECT_ID\"}"
N.B. For JSON-escaping strings with standard tools, take a look at function json_stringify in awk

jq format when running from a bash script with variable expansion

I've got a jq command that works when running directly from the shell or from within a shell script, but when I try to add variable expansion, I get jq errors for unexpected format or invalid characters. My goal is to have a quick and easy way to update some json configuration.
Here's a simplified example.
The format of the json I'm modifying:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
}
The jq without variable expansion:
cat myfile.json | jq '.pets."some-new-pet" += {PetInfo: {name: "my-brand-new-pet"}, toys: ["toy1", "toy2", "toy3"]}}'
The above runs fine, and adds the new pets.some-new-pet entry to my json.
Below is what I'm trying to do with variable expansion that fails.
jq_args = "'.pets.\"${PET}\" += {PetInfo: {name: \"${NAME}\"}, toys: [\"${toy1}-postfix\", \"${toy2}-postfix\", \"${toy3}-postfix\"]}}'"
cat myfile.json | jq $jq_args
The error message I get with the above:
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1: '.pets."some-new-pet"
My file is formatted as utf-8 and uses LF line endings.
I do not recommend constructing a jq filter using variable expansion or printf. It will work for simple cases but will fail if the string contains double quotes, backslashes or control-codes, as they have special meanings inside a JSON string. As an alternative to using printf, jq has a way to pass in variables directly via the command-line, avoiding all these issues.
pet='some-second-pet'
name='my-even-newer'
toy1=toy1
toy2=toy2
toy3=toy3
jq \
--arg pet "$pet" \
--arg name "$name" \
--arg toy1 "$toy1" \
--arg toy2 "$toy2" \
--arg toy3 "$toy3" \
'.pets[$pet] += {
PetInfo: {name: $name},
toys: ["\($toy1)-postfix", "\($toy2)-postfix", "\($toy3)-postfix"]
}' \
myfile.json
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet",
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
},
"some-second-pet": {
"PetInfo": {
"name": "my-even-newer-pet"
},
"toys": [
"toy1-postfix",
"toy2-postfix",
"toy3-postfix"
]
}
}
}
It would be cleaner and less error prone to format the string using printf
PET='dog'
NAME='sam'
toy1="t1"
toy2="t2"
toy3="t3"
jq_args=$(printf '.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s-postfix", "%s-postfix", "%s-postfix"]}}' "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}")
echo "$jq_args"
Result:
.pets."dog" += {PetInfo: {name: "sam"}, toys: ["t1-postfix", "t2-postfix", "t3-postfix"]}
Additionally, redundant quoting could be avoided by quoting the arg on this command
cat myfile.json | jq "$jq_args"
Fix your jq code by removing extra } at end
Fix bash jq call:
Add cotes "..." around your $jq_args
so don't use singles '...' in your jq_args definition
Use printf with -v option to define jq_args:
printf -v jq_args "...format..." value1 value2 ...
So your code became:
PET="some-new-pet"
NAME="my-brand-new-pet"
toy1="toy1"
toy2="toy2"
toy3="toy3"
format='.pets."%s" += {PetInfo: {name: "%s"}, toys: ["%s", "%s", "%s"]}'
printf -v jq_args "${format}" "${PET}" "${NAME}" "${toy1}" "${toy2}" "${toy3}"
cat myfile.json | jq "$jq_args"
Output:
{
"pets": {
"some-new-pet": {
"PetInfo": {
"name": "my-brand-new-pet"
},
"toys": [
"toy1",
"toy2",
"toy3"
]
}
}
}
Notes:
When you define your format, you put it into simple cotes '...'. It's really better to format JSON (or XML) without back-slashes (\\) before each double cotes (")
Use printf -v variable_name. It's more readable than var_name=$(printf ...)
By constructing the jq filter ("code") using outer bash variables ("data") you may run into escaping issues, which could eventually break or even divert your filter. (see https://en.wikipedia.org/wiki/Code_injection)
Instead, use mechanisms by jq to introduce external data through variables (parameter --arg):
jq --arg pet "${PET}" \
--arg name "${NAME}" \
--arg toy1 "${toy1}-postfix" \
--arg toy2 "${toy2}-postfix" \
--arg toy3 "${toy3}-postfix" \
'
.pets[$pet] += {PetInfo: {$name, toys: [$toy1,$toy2,$toy3]}}
' myfile.json
If you have an unknown number of variables to include, check out jq's --args parameter (note the additional s)

Unable to loop through the JSON internal Array having spaces in values using Bash script JQ [duplicate]

Background
I want to be able to pass a json file to WP CLI, to iteratively create posts.
So I thought I could create a JSON file:
[
{
"post_type": "post",
"post_title": "Test",
"post_content": "[leaflet-map][leaflet-marker]",
"post_status": "publish"
},
{
"post_type": "post",
"post_title": "Number 2",
"post_content": "[leaflet-map fitbounds][leaflet-circle]",
"post_status": "publish"
}
]
and iterate the array with jq:
cat posts.json | jq --raw-output .[]
I want to be able to iterate these to execute a similar function:
wp post create \
--post_type=post \
--post_title='Test Map' \
--post_content='[leaflet-map] [leaflet-marker]' \
--post_status='publish'
Is there a way I can do this with jq, or similar?
The closest I've gotten so far is this:
> for i in $(cat posts.json | jq -c .[]); do echo $i; done
But this seems to take issue with the (valid) spaces in the strings. Output:
{"post_type":"post","post_title":"Test","post_content":"[leaflet-map][leaflet-marker]","post_status":"publish"}
{"post_type":"post","post_title":"Number
2","post_content":"[leaflet-map
fitbounds][leaflet-circle]","post_status":"publish"}
Am I way off with this approach, or can it be done?
Use a while to read entire lines, rather than iterating over the words resulting from the command substitution.
while IFS= read -r obj; do
...
done < <(jq -c '.[]' posts.json)
Maybe this would work for you:
Make a bash executable, maybe call it wpfunction.sh
#!/bin/bash
wp post create \
--post_type="$1"\
--post_title="$2" \
--post_content="$3" \
--post_status="$4"
Then run jq on your posts.json and pipe it into xargs
jq -M -c '.[] | [.post_type, .post_title, .post_content, .post_status][]' \
posts.json | xargs -n4 ./wpfunction`
I am experimenting to see how this would handle post_content that contained quotes...
First generate an array of the arguments you wish to pass then convert to a shell compatible form using #sh. Then you could pass to xargs to invoke the command.
$ jq -r '.[] | ["post", "create", (to_entries[] | "--\(.key)=\(.value|tojson)")] | #sh' input.json | xargs wp

Replace text with special characters using sed

I have a json string like this,
rssample='{ "Changes": [{"Action": "UPSERT","ResourceRecordSet": {"ResourceRecords":[{ "Value":""}], "Type": "TXT","NAME":"","TTL": 300}}]}'
I want to update the NAME and Value in it using the below variables..
name="testname"
newvalue="heritage=external-dns,external-dns/owner=us-east-1:sandbox-newtestowner,external-dns/resource=ingress/monitoring/prometheus-operator-alertmanager"
So I tried using sed,
newrs=$(sed -E "s/"NAME":""/"NAME":"$name"/g" <<< "$rssample")
newrs1=$(sed -E "s/"Value":""/"Value":"\$newvalue"/g" <<< "$newrs")
I am expecting below as output,
{ "Changes": [{"Action": "UPSERT","ResourceRecordSet": {"ResourceRecords":[{ "Value":"\"heritage=external-dns,external-dns/owner=us-east-1:sandbox-newtestowner,external-dns/resource=ingress/monitoring/prometheus-operator-alertmanager\""}], "Type": "TXT","NAME":"testname","TTL": 300}}]}
But I am empty Name.And getting error for value as,
sed: 1: "s/Value:/Value:"heritag ...": bad flag in substitute command: 'o'
My output is,
{ "Changes": [{"Action": "UPSERT","ResourceRecordSet": {"ResourceRecords":[{ "Value":""}], "Type": "TXT","NAME":"","TTL": 300}}]}
Please let me know how to fix this? is using sed good idea or jq?
It's generally safer, and therefore often better, to use a JSON-aware tool rather than sed when editing JSON. Using jq, one possibility would be:
jq --arg name "$name" --arg newvalue "$newvalue" '
.Changes[0].ResourceRecordSet |=
(.NAME=$name
| .ResourceRecords[0].Value = $newvalue)' <<< "$rssample"
A free-form approach
jq --arg name "$name" --arg newvalue "$newvalue" '
walk(if type == "object"
then if has("NAME") then .NAME=$name else . end
| if has("Value") then .Value = $newvalue else . end
else . end)' <<< "$rssample"

ShellScript to send Mattermost notification is not working

I want to send a Message into a Mattermost channel with the help of a ShellScript/WebHook/cURL. The following code is the function to send the Message.
function matterSend() {
ENDPOINT=https://url.to.Mattermost/WebhookID
USERNAME="${USER}"
PAYLOAD=$(cat <<'EOF'
'payload={
"username" : "${USERNAME}",
"channel" : "TestChannel",
"text" : "#### Test to \n
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
}'
EOF
)
echo "CURL: curl -i -X POST -d ${PAYLOAD} ${ENDPOINT}"
curl -i -X POST -d "${PAYLOAD}" "${ENDPOINT}"
}
As you can see, when I ECHO the command I get:
curl -i -X POST -d 'payload={
"username" : "TestUser",
"channel" : "TestChannel",
"text" : "#### Test to \n
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
}' https://url.to.Mattermost/WebhookID
If I paste that code directly into the terminal and execute it, it works. But when I run the script with the help of a Jenkins-Job I get the Error:
Unable to parse incoming data","message":"Unable to parse incoming
data".
Why is it not working?
Without knowledge of the API you are connecting to, I would guess that you need
# Drop function keyword, indent body
matterSend() {
# Lowercase variable names; declare them local
local endpoint=https://url.to.Mattermost/WebhookID
local username=$USER
# Pro tip: don't use a variable for the payload if it's effectively static
payload=$(cat <<-__EOF
payload={
"username" : "$username",
"channel" : "TestChannel",
"text" : "#### Test to \\n| TestR | TestS | New Mode |\\n|:-----------|:-----------:|-----------------------------------------------:|\\n| ${2} | ${3} | ${1} :white_check_mark: |\\n"
}
__EOF
)
echo "CURL: curl -i -X POST -d $payload $endpoint"
curl -i -X POST -d "$payload" "$endpoint"
}
Replacing the newlines inside the "text" element with \n (and doubling the backslash because the here document is now being interpreted by the shell when it's assigned) is mildly speculative; perhaps remove the remaining newlines, too. The real beef is removing the misplaced literal single quotes around the payload.
Maybe also explore printf for formatting fixed-width tabular text.
The here document's <<-__EOF uses the unquoted separator __EOF and the dash before it says to remove any tabs from the beginning of each line. Needless to say, the indentation on those lines consists of a literal tab character.
Generating JSON (or XML, or other structured formats) via string concatenation leads to pain and suffering. Instead, use a tool that actually understands the format.
Using a compliant generator such as jq means you no longer need to be responsible for putting \ns in the data (as multi-character strings), changing "s in text to \", or any of the other otherwise-necessary munging.
matterSend() {
# Lowercase variable names; declare them local
local endpoint=https://url.to.Mattermost/WebhookID
local username=$USER
local text="#### Test to
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
payload=$(jq --arg username "$username" \
--arg channel "TestChannel" \
--arg text "$text" \
'{"username": $username, "channel": $channel, "text": $text}')
# Advice: use "set -x" if you want to trace the commands your script would run.
# ...or at least printf %q, as below; avoids misleading output from echo.
# printf '%q ' curl -i -X POST -d "$payload" "$endpoint" >&2; echo >&2
curl -i -X POST -d "$payload" "$endpoint"
}

Resources