I am using the bash code below to store the result of a curl command in a text file.
cat /c/customer_files/Bain/artifacts1.txt
sample=$(curl -X GET --header 'Accept: application/json' --header 'Authorization: Bearer <my_token>' '<api_>' | jq -r '.items[] | .id')
echo "$sample" >> /c/my_files/artifacts1.txt
This generates a text file with content below:
606b69cff140fe0d98e78d2a
60a40910c403d464225343b5
607f1e14d514043adcf4a0f6
60c36c380093aa519b816554
Now, I want to iterate through this file line by line. I am using the code below to do that.
while read -r line; do
#reading each line
echo "Line No. $n : $line";
n=$((n+1))
This is producing correct result as expected.
Line No. 1 : 606b69cff140fe0d98e78d2a
Line No. 2 : 60a40910c403d464225343b5
Line No. 3 : 607f1e14d514043adcf4a0f6
Line No. 4 : 60c36c380093aa519b816554
I want to pass variable $line to CURL command via json body below.
input_json="{"executable": "<some ID>", "keepTargetResources": true,"keepTargetRunProfiles": true,"advanced": {"artifactId": "$line","artifactType": "ACTION"}}"
However, this produces a result below:
,artifactType: ACTION}}d6a50bb75bbe81, keepTargetResources: true,keepTargetRunProfiles:
true,advanced: {artifactId: 606b69cff140fe0d98e78d2a
,artifactType: ACTION}}d6a50bb75bbe81, keepTargetResources: true,keepTargetRunProfiles:
true,advanced: {artifactId: 60a40910c403d464225343b5
,artifactType: ACTION}}d6a50bb75bbe81, keepTargetResources: true,keepTargetRunProfiles:
true,advanced: {artifactId: 607f1e14d514043adcf4a0f6
{executable: 60ca3bf02ed6a50bb75bbe81, keepTargetResources: true,keepTargetRunProfiles:
true,advanced: {artifactId: 60c36c380093aa519b816554,artifactType: ACTION}}
It creates an output in the desired format only for the last record:
{executable: 60ca3bf02ed6a50bb75bbe81, keepTargetResources:
true,keepTargetRunProfiles:true,advanced: {artifactId:
60c36c380093aa519b816554,artifactType: ACTION}}
For 1st 3 records, it looks like it is overwriting content at the start of the line.
What am I doing wrong? Please advice.
Thanks in Advance.
JSON needs to have its variables and values surrounded with ". Use \ to prevent the shell to interpret the ".
The extra carriage return is a mystery...
Maybe an extra clean up of line could make it
Prefer printf to echo and use "${line}" for more safety.
Give this a try:
artifact_id="$(printf "%s" "${line}" | sed 's/^\(.*[^[:blank:]]\)[[:blank:]]*$/\1/g')"
input_json="{\"executable\": \"<some ID>\", \"keepTargetResources\": true,\"keepTargetRunProfiles\": true,\"advanced\": {\"artifactId\": \"${artifact_id}\",\"artifactType\": \"ACTION\"}}"
I'm using git, then posting the commit message and other bits as a JSON payload to a server.
Currently I have:
MSG=`git log -n 1 --format=oneline | grep -o ' .\+'`
which sets MSG to something like:
Calendar can't go back past today
then
curl -i -X POST \
-H 'Accept: application/text' \
-H 'Content-type: application/json' \
-d "{'payload': {'message': '$MSG'}}" \
'https://example.com'
My real JSON has another couple of fields.
This works fine, but of course when I have a commit message such as the one above with an apostrophe in it, the JSON is invalid.
How can I escape the characters required in bash? I'm not familiar with the language, so am not sure where to start. Replacing ' with \' would do the job at minimum I suspect.
jq can do this.
Lightweight, free, and written in C, jq enjoys widespread community support with over 15k stars on GitHub. I personally find it very speedy and useful in my daily workflow.
Convert string to JSON
echo -n '猫に小判' | jq -Rsa .
# "\u732b\u306b\u5c0f\u5224"
To explain,
-R means "raw input"
-s means "include linebreaks" (mnemonic: "slurp")
-a means "ascii output" (optional)
. means "output the root of the JSON document"
Git + Grep Use Case
To fix the code example given by the OP, simply pipe through jq.
MSG=`git log -n 1 --format=oneline | grep -o ' .\+' | jq -Rsa .`
Using Python:
This solution is not pure bash, but it's non-invasive and handles unicode.
json_escape () {
printf '%s' "$1" | python -c 'import json,sys; print(json.dumps(sys.stdin.read()))'
}
Note that JSON is part of the standard python libraries and has been for a long time, so this is a pretty minimal python dependency.
Or using PHP:
json_escape () {
printf '%s' "$1" | php -r 'echo json_encode(file_get_contents("php://stdin"));'
}
Use like so:
$ json_escape "ヤホー"
"\u30e4\u30db\u30fc"
Instead of worrying about how to properly quote the data, just save it to a file and use the # construct that curl allows with the --data option. To ensure that the output of git is correctly escaped for use as a JSON value, use a tool like jq to generate the JSON, instead of creating it manually.
jq -n --arg msg "$(git log -n 1 --format=oneline | grep -o ' .\+')" \
'{payload: { message: $msg }}' > git-tmp.txt
curl -i -X POST \
-H 'Accept: application/text' \
-H 'Content-type: application/json' \
-d #git-tmp.txt \
'https://example.com'
You can also read directly from standard input using -d #-; I leave that as an exercise for the reader to construct the pipeline that reads from git and produces the correct payload message to upload with curl.
(Hint: it's jq ... | curl ... -d#- 'https://example.com' )
I was also trying to escape characters in Bash, for transfer using JSON, when I came across this. I found that there is actually a larger list of characters that must be escaped – particularly if you are trying to handle free form text.
There are two tips I found useful:
Use the Bash ${string//substring/replacement} syntax described in this thread.
Use the actual control characters for tab, newline, carriage return, etc. In vim you can enter these by typing Ctrl+V followed by the actual control code (Ctrl+I for tab for example).
The resultant Bash replacements I came up with are as follows:
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\\/\\\\} # \
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\//\\\/} # /
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\'/\\\'} # ' (not strictly needed ?)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//\"/\\\"} # "
JSON_TOPIC_RAW=${JSON_TOPIC_RAW// /\\t} # \t (tab)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//
/\\\n} # \n (newline)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^M/\\\r} # \r (carriage return)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^L/\\\f} # \f (form feed)
JSON_TOPIC_RAW=${JSON_TOPIC_RAW//^H/\\\b} # \b (backspace)
I have not at this stage worked out how to escape Unicode characters correctly which is also (apparently) required. I will update my answer if I work this out.
OK, found out what to do. Bash supports this natively as expected, though as always, the syntax isn't really very guessable!
Essentially ${string//substring/replacement} returns what you'd image, so you can use
MSG=${MSG//\'/\\\'}
To do this. The next problem is that the first regex doesn't work anymore, but that can be replaced with
git log -n 1 --pretty=format:'%s'
In the end, I didn't even need to escape them. Instead, I just swapped all the ' in the JSON to \". Well, you learn something every day.
git log -n 1 --format=oneline | grep -o ' .\+' | jq --slurp --raw-input
The above line works for me. refer to
https://github.com/stedolan/jq for more jq tools
I found something like that :
MSG=`echo $MSG | sed "s/'/\\\\\'/g"`
The simplest way is using jshon, a command line tool to parse, read and create JSON.
jshon -s 'Your data goes here.' 2>/dev/null
[...] with an apostrophe in it, the JSON is invalid.
Not according to https://www.json.org. A single quote is allowed in a JSON string.
How can I escape the characters required in bash?
You can use xidel to properly prepare the JSON you want to POST.
As https://example.com can't be tested, I'll be using https://api.github.com/markdown (see this answer) as an example.
Let's assume 'çömmít' "mêssågè" as the exotic output of git log -n 1 --pretty=format:'%s'.
Create the (serialized) JSON object with the value of the "text"-attribute properly escaped:
$ git log -n 1 --pretty=format:'%s' | \
xidel -se 'serialize({"text":$raw},{"method":"json","encoding":"us-ascii"})'
{"text":"'\u00E7\u00F6mm\u00EDt' \"m\u00EAss\u00E5g\u00E8\""}
Curl (variable)
$ eval "$(
git log -n 1 --pretty=format:'%s' | \
xidel -se 'msg:=serialize({"text":$raw},{"method":"json","encoding":"us-ascii"})' --output-format=bash
)"
$ echo $msg
{"text":"'\u00E7\u00F6mm\u00EDt' \"m\u00EAss\u00E5g\u00E8\""}
$ curl -d "$msg" https://api.github.com/markdown
<p>'çömmít' "mêssågè"</p>
Curl (pipe)
$ git log -n 1 --pretty=format:'%s' | \
xidel -se 'serialize({"text":$raw},{"method":"json","encoding":"us-ascii"})' | \
curl -d#- https://api.github.com/markdown
<p>'çömmít' "mêssågè"</p>
Actually, there's no need for curl if you're already using xidel.
Xidel (pipe)
$ git log -n 1 --pretty=format:'%s' | \
xidel -s \
-d '{serialize({"text":read()},{"method":"json","encoding":"us-ascii"})}' \
"https://api.github.com/markdown" \
-e '$raw'
<p>'çömmít' "mêssågè"</p>
Xidel (pipe, in-query)
$ git log -n 1 --pretty=format:'%s' | \
xidel -se '
x:request({
"post":serialize(
{"text":$raw},
{"method":"json","encoding":"us-ascii"}
),
"url":"https://api.github.com/markdown"
})/raw
'
<p>'çömmít' "mêssågè"</p>
Xidel (all in-query)
$ xidel -se '
x:request({
"post":serialize(
{"text":system("git log -n 1 --pretty=format:'\''%s'\''")},
{"method":"json","encoding":"us-ascii"}
),
"url":"https://api.github.com/markdown"
})/raw
'
<p>'çömmít' "mêssågè"</p>
This is an escaping solution using Perl that escapes backslash (\), double-quote (") and control characters U+0000 to U+001F:
$ echo -ne "Hello, 🌵\n\tBye" | \
perl -pe 's/(\\(\\\\)*)/$1$1/g; s/(?!\\)(["\x00-\x1f])/sprintf("\\u%04x",ord($1))/eg;'
Hello, 🌵\u000a\u0009Bye
I struggled with the same problem. I was trying to add a variable on the payload of cURL in bash and it kept returning as invalid_JSON. After trying a LOT of escaping tricks, I reached a simple method that fixed my issue. The answer was all in the single and double quotes:
curl --location --request POST 'https://hooks.slack.com/services/test-slack-hook' \
--header 'Content-Type: application/json' \
--data-raw '{"text":'"$data"'}'
Maybe it comes in handy for someone!
I had the same idea to send a message with commit message after commit.
First i tryed similar was as autor here.
But later found a better and simpler solution.
Just created php file which is sending message and call it with wget.
in hooks/post-receive :
wget -qO - "http://localhost/git.php"
in git.php:
chdir("/opt/git/project.git");
$git_log = exec("git log -n 1 --format=oneline | grep -o ' .\+'");
And then create JSON and call CURL in PHP style
Integrating a JSON-aware tool in your environment is sometimes a no-go, so here's a POSIX solution that should work on every UNIX/Linux:
json_stringify() {
[ "$#" -ge 1 ] || return 1
LANG=C awk '
BEGIN {
for ( i = 1; i <= 127; i++ )
repl[ sprintf( "%c", i) ] = sprintf( "\\u%04x", i )
for ( i = 1; i < ARGC; i++ ) {
s = ARGV[i]
printf("%s", "\"")
while ( match( s, /[\001-\037\177"\\]/ ) ) {
printf("%s%s", \
substr(s,1,RSTART-1), \
repl[ substr(s,RSTART,RLENGTH) ] \
)
s = substr(s,RSTART+RLENGTH)
}
print s "\""
}
exit
}
' "$#"
}
Or using the widely available perl:
json_stringify() {
[ "$#" -ge 1 ] || return 1
LANG=C perl -le '
for (#ARGV) {
s/[\x00-\x1f\x7f"\\]/sprintf("\\u%04x",ord($0))/ge;
print "\"$_\""
}
' -- "$#"
}
Then you can do:
json_stringify '"foo\bar"' 'hello
world'
"\u0022foo\bar\u0022"
"hello\u000aworld"
limitations:
Doesn't handle NUL bytes.
Doesn't validate the input for UNICODE, it only escapes the mandatory ASCII characters specified in the RFC 8259.
Replying to OP's question:
MSG=$(git log -n 1 --format=oneline | grep -o ' .\+')
curl -i -X POST \
-H 'Accept: application/text' \
-H 'Content-type: application/json' \
-d '{"payload": {"message": '"$(json_stringify "$MSG")"'}}' \
'https://example.com'
When I'm trying to execute script below and getting error: "curl: (3) URL using bad/illegal format or missing URL"
#!/bin/bash
stage="develop"
branch="branch_name"
getDefinition=$(curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions?api-version=5.1")
for def in $(echo "$getDefinition" | jq '.value[] | select (.path=="\\Some_path\\'$stage'") | .id'); do
getBuildInfo=$(curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1")
# echo $def
body=$(echo "${getBuildInfo}" | jq '.repository.defaultBranch = "refs/heads/release/'"${branch}"'"' | jq '.options[].inputs.branchFilters = "[\"+refs/heads/release/'"${branch}"'\"]"' | jq '.triggers[].branchFilters[] = "+refs/heads/release/'"${branch}"'"')
echo ${body} > data.json
done
It happens when I'm trying to pass variable ${def} into a line:
curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1"
But when I declare an array, curl works as expected.
Example:
declare -a def
def=(1 2 3 4)
curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1"
Could you please suggest how can I pass variable into URL properly?
Do you need to call curl for 4 times? If so.
for def in 1 2 3 4; do curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1"; done
Set IFS=$' \t\r\n' at the top of your script.
IFS is the Interactive Field Separator.
In UNIX, IFS is space, tab, newline, or $' \t\n', but on Windows this needs to be $' \t\r\n'.
The ^M character is \r.
I used the jq built-in method select to parse the json string in shell script, and got an error: command not found.
Here is my shell script: test.sh
#!/bin/bash
function test(){
json='[{"id":1,"name":"jdjson"},{"id":2,"name":"imagetookit"}]'
detail=`echo $json | jq .[]|select\(.id==2\)`
}
test
I just ran the script on the command line and got the following error:
$ bash test.sh
test.sh:行5: select(.id==2): 未找到命令 (means "command not found")
the select is built-in method of jq command, and I don't know why
Does the function need to be imported? How to import method of jq?
It's not a jq problem, it's a shell quoting issue:
mytest(){
json='[{"id":1,"name":"jdjson"},{"id":2,"name":"imagetookit"}]'
detail=$(echo "$json" | jq '.[]|select(.id==2)')
echo "$detail"
}
mytest
{
"id": 2,
"name": "imagetookit"
}
Notice the single quote ' around the jq command and so no need for backslash \.
Also prefer the $(...) instead of the old backtick notation.
I have problem with assigning curl as variable and assign curl's output to variable:
#get results url, format json
URL=$(curl https://api.apifier.com/xy)
#jq is a cli json interpreter
#resultUrl contains the final URL which we want download
OK= "$URL" | jq '.resultsUrl'
#api probably is running
sleep 5
curl "$OK"
Maybe it is trivial, but I don't know where is the problem.
My guess is:
jq '.resultsUrl'
outputs the field resultsUrl with quotes, so curl does not process it correctly. Furthermore, $URL | ... does not work, you would have to use echo or curl directly.
Try
OK=$(curl -s https://api.apifier.com/v1/xHbBnrZ9rxF4CdKjo/crawlers/Example_Alcatraz_Cruises/execute?token=nJ9ohCHZPaJRFEb7nFqtzm76u | jq -r '.resultsUrl')
curl -s "$OK"
which results for me in
[{ "id": 2, "url": "https://www.alcatrazcruises.com/SearchEventDaySpan.aspx?date=02-25-2016&selected=", "loadedUrl": "https://www.alcatrazcruises.com/SearchEventDaySpan.aspx?date=02-25-2016&selected=", "requestedAt": "2016-02-25T23:24:52.611Z", "loadingStartedAt": "2016-02-25T23:24:54.663Z", "loadingFinishedAt": "2016-02-25T23:24:55.642Z", "loadErrorCode": null, "pageFunctionStartedAt": "2016-02-25T23:24:55.839Z", "pageFunctionFinishedAt": "2016-02-25T23:24:55.841Z", "uniqueKey": "https://www.alcatrazcruises.com/SearchEventDaySpan.aspx?date=02-25-2016&selected=", "type": "UserEnqueued", ...
This should be what you expect.
However, sometimes the first API call yields an error:
{
"type": "ALREADY_RUNNING",
"message": "The act is already running and concurrent execution is not allowed"
}
so resultsURL will be null, you will have to handle this error case.
Your line
OK= "$URL" | jq '.resultsURL'
sets the environment variable OK to an empty string, then tries to execute "$URL" as a command and pipe its output to jq. If you want to setOK to the result of a command, you have to use $OK=(...), just like you did when setting URL. The correct syntax is:
OK=$(echo "$URL" | jq '.resultsURL')
And to remove the quotes from the output of .jq, you can do:
OK=$(echo "$URL" | jq '.resultsURL' | tr -d '"')