I have a curl request that accepts JSON as its payload. The JSON payload is a multi-line string. I'm having trouble piping the output of this curl after the EOF.
curl https://foo.bar/v1/baz \
-H "FOO-BAR: BAZ" \
-X POST -d #- <<'EOF'
{
"foo" : "foo_foo",
"bar": {}
}
EOF
This works, but if I want to pipe the output of this to something, for example python -m json.tool, I have a problem. The following doesn't work:
curl https://foo.bar/v1/baz \
-H "FOO-BAR: BAZ" \
-X POST -d #- <<'EOF'
{
"foo" : "foo_foo",
"bar": {}
}
EOF | python -m json.tool
The here document doesn't begin until the next (logical) line after the command itself. Like any other redirection operator, <<'EOF' does not need to be the final token on the command line.
curl https://foo.bar/v1/baz \
-H "FOO-BAR: BAZ" \
-X POST -d #- <<'EOF' | python -m json.tool
{
"foo" : "foo_foo",
"bar": {}
}
EOF
Related
Trying to send some data via API call (curl) but having issue with the format of the curl command:
serviceMsg="$(cat /tmp/response_time)"
perfData=$(/bin/echo "${serviceMsg}" | /bin/sed 's/,//g')
StatusCode=0
curl -k -s -v -u user:password -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/actions/process-check-result?service=ExternalURL!ResponseTime' -d "{ \"exit_status\": $StatusCode, \"plugin_output\": \"${perfData}\", \"performance_data\": [ ${perfData} ]}"
Here is the error I am getting:
* Connection #0 to host localhost left intact
{
"error": 400.0,
"status": "Invalid request body: Error: lexical error: invalid character inside string.\n { \"exit_status\": 0, \"plugin_ou\n (right here) ------^\n\n"
}
what I am missing ?
Don't hand-code JSON: use jq to generate it for you:
perfData=$(sed 's/,//g' /tmp/response_time)
json=$(jq -n --arg sc "$StatusCode" --arg pd "$perfData" \
'{exit_status: $sc, plugin_output: $pd, performance_data: [ $pd ]}')
url='https://localhost:5665/v1/actions/process-check-result?service=ExternalURL!ResponseTime'
curl -ksv -u user:password -H 'Accept: application/json' -X POST -d "$json" "$url"
I have a code which returns an access token string.
#!/bin/bash
token=$(curl --globoff -X POST "https://example.com/token" -F 'grant_type=client_credentials' -F 'client_id=d86fc9b690963b8dda602bd26f33454r457e9024a4ecccf4c3bbf66ffcbc241b' -F 'client_code'='ff148c56118728b62c9f2ew3e4r009a7a1c645276b70611fa32fa055b9944934')
echo "$token" > Token.txt
The output given by this command is :
{
"access_token": "5048e7d38e73b4a809a4fcb219b63ae34f34f43f83d6663ffd777203afb5654ab",
"token_type": "bearer",
"expires_in": 7200
}
ie. the variable token contains the above result. My question is how to get the access token 5048e7d38e73b4a809a4fcb219b63ae34f34f43f83d6663ffd777203afb5654ab alone from this and save to another variable.
First try jq, if not possible, this is a workaround using awk:
access_token=$(curl --globoff -X POST "https://example.com/token" \
-F 'grant_type=client_credentials' \
-F 'client_id=d86fc9b690963b8dda602bd26f33454r457e9024a4ecccf4c3bbf66ffcbc241b' \
-F 'client_code'='ff148c56118728b62c9f2ew3e4r009a7a1c645276b70611fa32fa055b9944934'|\
awk -F\" '/access_token/{print $4}')
The way to do this in jq would be to do
access_token=$(curl --globoff -X POST "https://example.com/token" \
-F 'grant_type=client_credentials' \
-F 'client_id=d86fc9b690963b8dda602bd26f33454r457e9024a4ecccf4c3bbf66ffcbc241b' \
-F 'client_code'='ff148c56118728b62c9f2ew3e4r009a7a1c645276b70611fa32fa055b9944934'|\
jq -r '.access_token' )
I am using this in bash to get and use the token ...
access_tok=$(curl -s -X POST -d "client_id=$(< "$LOGON_HOME/client_id" )&client_secret=$(< "$LOGON_HOME/client_secret")&grant_type=client_credentials" "${TOKEN_URL}" )
access_tok=${access_tok#*access_token\":\"}
access_tok=${access_tok%%\"*}
and later
curl -l -s -O -H "Authorization: Bearer ${access_tok}" "${FILE_URL}"
This is the script i used to export some documents to elasticsearch but no luck
#!/bin/ksh
set -v
trap read debug
date=$(date +%Y-%m-%d);
echo $date;
config_file="/home/p.sshanm/reports_elastic.cfg";
echo $config_file;
URL="http://p-acqpes-app01.wirecard.sys:9200/reports-"$date"";
echo $URL;
find /transfers/documents/*/done/ -type f -name "ABC-Record*_${date}*.csv"|
while IFS='' read -r -d '' filename
do
echo "filename : ${filename}"
var=$(base64 "$filename"| perl -pe 's/\n//g');
#if i use below it will fail as argument too long , so i used with curl option #
# var1= $(curl -XPUT 'http://localhost:9200/reports-'$date'/document/reports?pipeline=attachment&pretty' -d' { "data" : "'$var'" }')
var1=$(curl -X PUT -H "Content-Type: application/json" -d #- "$URL" >>CURLDATA
{ "data": "$var" }
CURL_DATA)
done;
If i use below it as
var1= $(curl -XPUT 'http://localhost:9200/reports-'$date'/document/reports?pipeline=attachment&pretty' -d' { "data" : "'$var'" }')
will fail as below, so i used with curl option #
argument too long
Your syntax to read from stdin is wrong, the here-doc string should have been (<<) and the de-limiters are mis-matching use CURL_DATA at both places.
curl -X PUT -H "Content-Type: application/json" -d #- "$URL" <<CURL_DATA
{ "data": "$var" }
CURL_DATA
I've got a shell script I've been using to post stuff to a hipchat channel. It works ok until I try and send a message that has characters that need escaping. I run the command like so (note the extra backslash in there to cause a problem)
/usr/local/bin/hipchatmsg.sh "my great message here \ " red
And my code in my bash script (hipchatmsg.sh) that matters is this:
# Make sure message is passed
if [ -z ${1+x} ]; then
echo "Provide a message to create the new notification"
exit 1
else
MESSAGE=$1
fi
// send locally via curl
/usr/bin/curl -H "Content-Type: application/json" \
-X POST \
-k \
-d "{\"color\": \"$COLOR\", \"message_format\": \"text\", \"message\": \"$MESSAGE\" }" \
$SERVER/v2/room/$ROOM_ID/notification?auth_token=$AUTH_TOKEN &
// $server and $room are defined earlier
exit 0
If I try and run the command above with any characters that need escaping, I will get an error like this:
{
"error": {
"code": 400,
"message": "The request body cannot be parsed as valid JSON: Invalid \\X escape sequence u'\\\\': line 1 column 125 (char 124)",
"type": "Bad Request"
}
}
I found something kind of similar on here where the best advice was to try sending the curl post with --data-urlencode, so I tried like this:
/usr/bin/curl -H "Content-Type: application/json" \
-X POST \
-k \
-d --data-urlencode "{\"color\": \"$COLOR\", \"message_format\": \"text\", \"message\": \"$MESSAGE\" }" \
$SERVER/v2/room/$ROOM_ID/notification?auth_token=$AUTH_TOKEN &
But this had no effect.
What am I missing here?
The easiest thing to do is use a program like jq to generate the JSON; it will take care of escaping what needs to be escaped.
jq -n --arg color "$COLOR" \
--arg message "$MESSAGE" \
'{color: $color, message_format: "text", message: $message}' |
/usr/bin/curl -H "Content-Type: application/json" \
-X POST \
-k \
-d#- \
$SERVER/v2/room/$ROOM_ID/notification?auth_token=$AUTH_TOKEN &
The argument #- to -d tells curl to read from standard input, which is supplied from jq via the pipe. The --arg options to jq make available JSON-encoded strings to the filter, which is simply a JSON object expression.
Consider the curl command below, is it possible to allow newline in JSON (without the minify) and execute directly in bash (Mac/Ubuntu)
curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
-d \
'
{
"field1": "test",
"field2": {
"foo": "bar"
}
}'
When I run the command above, seems error occurred at the second {
How to fix the above command?
Updated: actually I was able to run the command without issue previously, not sure why problem happen recently.
I remembered another way to do this with a "Here Document" as described in the Bash man page and detailed here. The #- means to read the body from STDIN, while << EOF means to pipe the script content until "EOF" as STDIN to curl. This layout may be easier to read than using separate files or the "echo a variable" approach.
curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: application/json; charset=utf-8' \
--data-binary #- << EOF
{
"field1": "test",
"field2": {
"foo": "bar"
}
}
EOF
NOTE: Use the --trace <outfile> curl option to record exactly what goes over the wire. For some reason, this Here Document approach strips newlines. (Update: Newlines were stripped by curl -d option. Corrected!)
Along the lines of Martin's suggestion of putting the JSON in a variable, you could also put the JSON in a separate file, and then supply the filename to -d using curl's # syntax:
curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
-d #myfile.json
The disadvantage is obvious (2 or more files where you used to have one.) But on the plus side, your script could accept a filename or directory argument and you'd never need to edit it, just run it on different JSON files. Whether that's useful depends on what you are trying to accomplish.
For some reason, this Here Document approach strips newlines
#eric-bolinger the reason the Heredoc strips newlines is because you need to tell your Heredoc to preserve newlines by quoting the EOF:
curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
-d #- <<'EOF'
{
"field1": "test",
"field2": {
"foo": "bar"
}
}
EOF
Notice the single-ticks surrounding EOF the first time it's defined, but not the second.
You should use outer double quotes, and the escape all inner quotes like this:
curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
-d \
"
{
\"field1\": \"test\",
\"field2\": {
\"foo\": \"bar\"
}
}"
You could assign your json to a var:
json='
{
"field1": "test",
"field2": {
"foo": "bar"
}
}'
Now you can forward this to curl using stdin:
echo $json | curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
-d #-
I think this can be an answer
curl -0 -v -X POST http://www.example.com/api/users \
-H "Expect:" \
-H 'Content-Type: text/json; charset=utf-8' \
--data-raw '
{
"field1": "test",
"field2": {
"foo": "bar"
}
}'