I'm getting the code below
#!/bin/bash
input="processes.txt"
while IFS= read -r line
do
curl --location --request POST 'https://api.controllegal.zymdev.com/v2/getProcessesInfo' \
--header 'Content-Type: application/json' \
--data-raw '{
"data":{
"process_id": "$line",
"location_id":"11001",
"entity":"608-True-3110-11001"
}
}'
done < "$input"
from Postman to make a cURL POST request and using it in a bash script to loop a txt file line and send it in the data json raw, but for some reason it returns
false
each time the curl get executed
when doing
echo $line
it prints fine the line string which is a something like "11001010200020160112400"
and if I use the string directly in the data-raw it works fine.
I tried using double quotes like
""$line""
but it throws
Unexpected token $ in JSON
In bash, a pair of single-quotes (') prevents variable substitution inside. You would need to do something like
--data-raw "{
\"data\":{
\"process_id\": \"$line\",
...
"
so that the value of $line gets substituted.
I'm using git, then posting the commit message and other bits as a JSON payload to a server.
Currently I have:
MSG=`git log -n 1 --format=oneline | grep -o ' .\+'`
which sets MSG to something like:
Calendar can't go back past today
then
curl -i -X POST \
-H 'Accept: application/text' \
-H 'Content-type: application/json' \
-d "{'payload': {'message': '$MSG'}}" \
'https://example.com'
My real JSON has another couple of fields.
This works fine, but of course when I have a commit message such as the one above with an apostrophe in it, the JSON is invalid.
How can I escape the characters required in bash? I'm not familiar with the language, so am not sure where to start. Replacing ' with \' would do the job at minimum I suspect.
jq can do this.
Lightweight, free, and written in C, jq enjoys widespread community support with over 15k stars on GitHub. I personally find it very speedy and useful in my daily workflow.
Convert string to JSON
echo -n '猫に小判' | jq -Rsa .
# "\u732b\u306b\u5c0f\u5224"
To explain,
-R means "raw input"
-s means "include linebreaks" (mnemonic: "slurp")
-a means "ascii output" (optional)
. means "output the root of the JSON document"
Git + Grep Use Case
To fix the code example given by the OP, simply pipe through jq.
MSG=`git log -n 1 --format=oneline | grep -o ' .\+' | jq -Rsa .`
Using Python:
This solution is not pure bash, but it's non-invasive and handles unicode.
json_escape () {
printf '%s' "$1" | python -c 'import json,sys; print(json.dumps(sys.stdin.read()))'
}
Note that JSON is part of the standard python libraries and has been for a long time, so this is a pretty minimal python dependency.
Or using PHP:
json_escape () {
printf '%s' "$1" | php -r 'echo json_encode(file_get_contents("php://stdin"));'
}
Use like so:
$ json_escape "ヤホー"
"\u30e4\u30db\u30fc"
Instead of worrying about how to properly quote the data, just save it to a file and use the # construct that curl allows with the --data option. To ensure that the output of git is correctly escaped for use as a JSON value, use a tool like jq to generate the JSON, instead of creating it manually.
jq -n --arg msg "$(git log -n 1 --format=oneline | grep -o ' .\+')" \
'{payload: { message: $msg }}' > git-tmp.txt
curl -i -X POST \
-H 'Accept: application/text' \
-H 'Content-type: application/json' \
-d #git-tmp.txt \
'https://example.com'
You can also read directly from standard input using -d #-; I leave that as an exercise for the reader to construct the pipeline that reads from git and produces the correct payload message to upload with curl.
(Hint: it's jq ... | curl ... -d#- 'https://example.com' )
I was also trying to escape characters in Bash, for transfer using JSON, when I came across this. I found that there is actually a larger list of characters that must be escaped – particularly if you are trying to handle free form text.
There are two tips I found useful:
Use the Bash ${string//substring/replacement} syntax described in this thread.
Use the actual control characters for tab, newline, carriage return, etc. In vim you can enter these by typing Ctrl+V followed by the actual control code (Ctrl+I for tab for example).
The resultant Bash replacements I came up with are as follows:
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\\/\\\\} # \
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\//\\\/} # /
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\'/\\\'} # ' (not strictly needed ?)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\"/\\\"} # "
JSON_TOPIC_RAW=${JSON_TOPIC_RAW// /\\t} # \t (tab)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//
/\\\n} # \n (newline)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^M/\\\r} # \r (carriage return)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^L/\\\f} # \f (form feed)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^H/\\\b} # \b (backspace)
I have not at this stage worked out how to escape Unicode characters correctly which is also (apparently) required. I will update my answer if I work this out.
OK, found out what to do. Bash supports this natively as expected, though as always, the syntax isn't really very guessable!
Essentially ${string//substring/replacement} returns what you'd image, so you can use
MSG=${MSG//\'/\\\'}
To do this. The next problem is that the first regex doesn't work anymore, but that can be replaced with
git log -n 1 --pretty=format:'%s'
In the end, I didn't even need to escape them. Instead, I just swapped all the ' in the JSON to \". Well, you learn something every day.
git log -n 1 --format=oneline | grep -o ' .\+' | jq --slurp --raw-input
The above line works for me. refer to
https://github.com/stedolan/jq for more jq tools
I found something like that :
MSG=`echo $MSG | sed "s/'/\\\\\'/g"`
The simplest way is using jshon, a command line tool to parse, read and create JSON.
jshon -s 'Your data goes here.' 2>/dev/null
[...] with an apostrophe in it, the JSON is invalid.
Not according to https://www.json.org. A single quote is allowed in a JSON string.
How can I escape the characters required in bash?
You can use xidel to properly prepare the JSON you want to POST.
As https://example.com can't be tested, I'll be using https://api.github.com/markdown (see this answer) as an example.
Let's assume 'çömmít' "mêssågè" as the exotic output of git log -n 1 --pretty=format:'%s'.
Create the (serialized) JSON object with the value of the "text"-attribute properly escaped:
$ git log -n 1 --pretty=format:'%s' | \
xidel -se 'serialize({"text":$raw},{"method":"json","encoding":"us-ascii"})'
{"text":"'\u00E7\u00F6mm\u00EDt' \"m\u00EAss\u00E5g\u00E8\""}
Curl (variable)
$ eval "$(
git log -n 1 --pretty=format:'%s' | \
xidel -se 'msg:=serialize({"text":$raw},{"method":"json","encoding":"us-ascii"})' --output-format=bash
)"
$ echo $msg
{"text":"'\u00E7\u00F6mm\u00EDt' \"m\u00EAss\u00E5g\u00E8\""}
$ curl -d "$msg" https://api.github.com/markdown
<p>'çömmít' "mêssågè"</p>
Curl (pipe)
$ git log -n 1 --pretty=format:'%s' | \
xidel -se 'serialize({"text":$raw},{"method":"json","encoding":"us-ascii"})' | \
curl -d#- https://api.github.com/markdown
<p>'çömmít' "mêssågè"</p>
Actually, there's no need for curl if you're already using xidel.
Xidel (pipe)
$ git log -n 1 --pretty=format:'%s' | \
xidel -s \
-d '{serialize({"text":read()},{"method":"json","encoding":"us-ascii"})}' \
"https://api.github.com/markdown" \
-e '$raw'
<p>'çömmít' "mêssågè"</p>
Xidel (pipe, in-query)
$ git log -n 1 --pretty=format:'%s' | \
xidel -se '
x:request({
"post":serialize(
{"text":$raw},
{"method":"json","encoding":"us-ascii"}
),
"url":"https://api.github.com/markdown"
})/raw
'
<p>'çömmít' "mêssågè"</p>
Xidel (all in-query)
$ xidel -se '
x:request({
"post":serialize(
{"text":system("git log -n 1 --pretty=format:'\''%s'\''")},
{"method":"json","encoding":"us-ascii"}
),
"url":"https://api.github.com/markdown"
})/raw
'
<p>'çömmít' "mêssågè"</p>
This is an escaping solution using Perl that escapes backslash (\), double-quote (") and control characters U+0000 to U+001F:
$ echo -ne "Hello, 🌵\n\tBye" | \
perl -pe 's/(\\(\\\\)*)/$1$1/g; s/(?!\\)(["\x00-\x1f])/sprintf("\\u%04x",ord($1))/eg;'
Hello, 🌵\u000a\u0009Bye
I struggled with the same problem. I was trying to add a variable on the payload of cURL in bash and it kept returning as invalid_JSON. After trying a LOT of escaping tricks, I reached a simple method that fixed my issue. The answer was all in the single and double quotes:
curl --location --request POST 'https://hooks.slack.com/services/test-slack-hook' \
--header 'Content-Type: application/json' \
--data-raw '{"text":'"$data"'}'
Maybe it comes in handy for someone!
I had the same idea to send a message with commit message after commit.
First i tryed similar was as autor here.
But later found a better and simpler solution.
Just created php file which is sending message and call it with wget.
in hooks/post-receive :
wget -qO - "http://localhost/git.php"
in git.php:
chdir("/opt/git/project.git");
$git_log = exec("git log -n 1 --format=oneline | grep -o ' .\+'");
And then create JSON and call CURL in PHP style
Integrating a JSON-aware tool in your environment is sometimes a no-go, so here's a POSIX solution that should work on every UNIX/Linux:
json_stringify() {
[ "$#" -ge 1 ] || return 1
LANG=C awk '
BEGIN {
for ( i = 1; i <= 127; i++ )
repl[ sprintf( "%c", i) ] = sprintf( "\\u%04x", i )
for ( i = 1; i < ARGC; i++ ) {
s = ARGV[i]
printf("%s", "\"")
while ( match( s, /[\001-\037\177"\\]/ ) ) {
printf("%s%s", \
substr(s,1,RSTART-1), \
repl[ substr(s,RSTART,RLENGTH) ] \
)
s = substr(s,RSTART+RLENGTH)
}
print s "\""
}
exit
}
' "$#"
}
Or using the widely available perl:
json_stringify() {
[ "$#" -ge 1 ] || return 1
LANG=C perl -le '
for (#ARGV) {
s/[\x00-\x1f\x7f"\\]/sprintf("\\u%04x",ord($0))/ge;
print "\"$_\""
}
' -- "$#"
}
Then you can do:
json_stringify '"foo\bar"' 'hello
world'
"\u0022foo\bar\u0022"
"hello\u000aworld"
limitations:
Doesn't handle NUL bytes.
Doesn't validate the input for UNICODE, it only escapes the mandatory ASCII characters specified in the RFC 8259.
Replying to OP's question:
MSG=$(git log -n 1 --format=oneline | grep -o ' .\+')
curl -i -X POST \
-H 'Accept: application/text' \
-H 'Content-type: application/json' \
-d '{"payload": {"message": '"$(json_stringify "$MSG")"'}}' \
'https://example.com'
When I'm trying to execute script below and getting error: "curl: (3) URL using bad/illegal format or missing URL"
#!/bin/bash
stage="develop"
branch="branch_name"
getDefinition=$(curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions?api-version=5.1")
for def in $(echo "$getDefinition" | jq '.value[] | select (.path=="\\Some_path\\'$stage'") | .id'); do
getBuildInfo=$(curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1")
# echo $def
body=$(echo "${getBuildInfo}" | jq '.repository.defaultBranch = "refs/heads/release/'"${branch}"'"' | jq '.options[].inputs.branchFilters = "[\"+refs/heads/release/'"${branch}"'\"]"' | jq '.triggers[].branchFilters[] = "+refs/heads/release/'"${branch}"'"')
echo ${body} > data.json
done
It happens when I'm trying to pass variable ${def} into a line:
curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1"
But when I declare an array, curl works as expected.
Example:
declare -a def
def=(1 2 3 4)
curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1"
Could you please suggest how can I pass variable into URL properly?
Do you need to call curl for 4 times? If so.
for def in 1 2 3 4; do curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1"; done
Set IFS=$' \t\r\n' at the top of your script.
IFS is the Interactive Field Separator.
In UNIX, IFS is space, tab, newline, or $' \t\n', but on Windows this needs to be $' \t\r\n'.
The ^M character is \r.
When i run this code, i get
./file.sh: line 23: ((: j<: syntax error: operand expected (error token is "<"). The $orgs+$i.json in the below code will generate files if it consists data, otherwise thise file wont be generated. The orgs and repos are from the file which i passed. Once the json file is created with data, then i will need to pull some fields like commit, repo, author, file fields from that json file. I will attach the json file format from which i'm trying to pull values.
Json file data looks like in below image
enter image description here
#!/bin/bash
IFS=$IFS,
usr='*************'
pwd='*************'
url='*************'
curl --header "Content-Type: application/json" --request GET
"https://$usr:$pwd#$url" |grep login|cut -d ':' -f2|sed 's/,//g'|sed
's/"//g' >> MHEOrgs.txt
File=./MHEOrgs.txt
while read orgs; do
#orgs= $i
curl -s --header "Content-Type: application/json" --request GET --user
"$usr:$pwd" $url/$orgs/repos?page=[1-100] >> $orgs+Repos.json
jq -r '.[].name' $orgs+Repos.json >>$orgs+Repolist.json
user="https://$usr:$pwd#********/$orgs/"
repos=`cat $orgs+Repolist.json`
for i in $repos;
do
echo $user"$i".git
#Below commabd will perform scan operation on the githubrepos and
generates multiple json files
~/go/bin/gitleaks --repo=$user"$i" --report=$orgs+$i.json
#Now i wanted to get below values from each file generated above.
numsecs=$(jq ".|length" $orgs+$i.json)
for ((j=0;j<$numsecs;j++))
do
commit=$(jq -r ".[$j]|.commit" $orgs+$i.json)
author=$(jq -r ".[$j]|.author" $orgs+$i.json)
file=$(jq -r ".[$j]|.file" $orgs+$i.json)
repo=$(jq -r ".[$j]|.repo" $orgs+$i.json)
if [ "${commit}" != "null" ]; then
echo "\"$repo\"," "\"$file\"," "\"$author\"," "\"$commit\",">>
gitleaks-scan-results.csv
else
echo "No leaks found"
fi
done
done
done < $File
That happens if numsecs isn't actually set to a (single integer) number.
One way that can happen is if your JSON file doesn't have any records in it at all; or if it has more than one (which would mean you'd have several numbers in jq's output, which bash can't correctly parse to a single number).
Some Examples
The zero-object document case:
$ numsecs=$(jq '. | length' <<<'')
$ declare -p numsecs
declare -- numsecs=""
$ for ((j=0;j<$numsecs;j++)); do break; done
-bash: ((: j<: syntax error: operand expected (error token is "<")
The multi-object document case:
$ numsecs=$(jq '. | length' <<<$'[1,2]\n[3,4]')
$ declare -p numsecs
declare -- numsecs="2
2"
$ for ((j=0;j<$numsecs;j++)); do break; done
-bash: ((: j<2
2: syntax error in expression (error token is "2")
I want to use this command into a bash script where each time I will have a different array input containing the parameters?
knowing that I have an array (as input from the user) where each column contains "parameteri=valuei".
I want to get rid of the hardcoded aspect in introducing the name and the value of each parameter.
For instance, with this input:
"id=123,verbosity=high"
I will eventually get this final instruction:
curl -X POST JENKINS_URL/job/JOB_NAME/build \
--user USER:TOKEN \
--data-urlencode json='{"parameter": [{"name":"id", "value":"123"}, {"name":"verbosity", "value":"high"}]}'
What is a clean way to do so?
You can make it the sexy way, building the jsonParameters from specified key=value parameters:
#!/bin/bash
jsonParameters=""
while IFS=',' read -r -a parameterEntries; do
for parameterEntry in "${parameterEntries[#]}"; do
IFS='=' read -r key value <<< "$parameterEntry"
[ ! -z "$jsonParameters" ] && jsonParameters="$jsonParameters,"
jsonParameters="$jsonParameters {\"name\":\"$key\", \"value\": \"$value\"}"
done
done <<< "$#"
Explanations:
the first loop will create the array named parameterEntries, with all your specified parameters, each element will contain key=value
then, the second loop, which iterates on each element of this array, will extract key, and value of it
eventually, it is only syntax writting to get the JSON output you want
the [ ! -z "$jsonParameters" ] && jsonParameters="$jsonParameters," is just here to add a separating coma, only if there is more than one element
Then you simply have to use the $jsonParameters where you want:
curl -X POST JENKINS_URL/job/JOB_NAME/build \
--user USER:TOKEN \
--data-urlencode json="{\"parameter\": [$jsonParameters]}"