I'm using curl command inside logstash exec plugin to post a message to a stride group. The plugin documentation stated that they were using ruby system method to execute the command, so I'm trying to run it in my ruby IRB.
Escaping the double quotes with backslash character is giving error The request body cannot be parsed as valid JSON. Here is the full error
irb(main):050:0' --inf-ruby-2f3827a9-23243-13517-726000--
curl: (6) Couldn't resolve host 'first'
curl: (3) [globbing] unmatched close brace/bracket in column 9
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 165 0 87 100 78 60 54 0:00:01 0:00:01 --:--:-- 60
{
"statusCode": 400,
"message": "The request body cannot be parsed as valid JSON"
}=> true
I tried swapping double quotes with single quotes and using full double quotes everywhere. Nothing is working.
system("curl -X POST -H 'Content-Type: application/json' -H 'Authorization: Bearer blah-blah' -d '{\"body\":{\"version\":1,\"type\":\"doc\",\"content\":[{\"type\":\"paragraph\",\"content\":[{\"type\":\"text\",\"text\":\"My first message!\"}]}]}}' --url 'https://api.atlassian.com/site/blah-blah/conversation/blah-blah/message'")
Is there any way to make this work?
EDIT: I have tried running the cURL command in the terminal and it is working fine.
Besides the following answers on StackOverflow, NetHTTP has notoriously poor documentation but can be used to post what you're interested in.
Related
Figuring out a new API.
I’m trying to call an endpoint and populate a JSON file with the data. I see the following output:
admin#server:~$ curl -H "Authorization: Bearer <NOT DISPLAYED FOR THIS POST>" -o /home/admin/result.json https://www.endpoint.com/manage/query/run?id=55408&cmd=service&output=json
[1] 14493
[2] 14494
admin#server:~$ % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
It hangs here seemingly indefinitely until I hit enter and then the below displays
[1]- Done curl -H "Authorization: Bearer <NOT DISPLAYED FOR THIS POST>" -o /home/admin/result.json https://www.endpoint.com/manage/query/run?id=55408
[2]+ Done cmd=service
No error messages, and no data in result.json. Calling the same cURL command without the -o option also returns the same results, when normally I would expect to see the data pop up in my terminal. If I visit the endpoint URL in browser (auth token can be a URL parameter as well for this API), I see the exact data I want to download. Changing the Auth Token makes no difference in the output.
I know every API is different, and there are a hundred different troubleshooting questions I haven't addressed in this post, but has anyone experienced this type of output before with a cURL command? I've never seen this behavior before.
The API is for Slate, an SIS for universities, if that helps. Thank you!
Your URL has a & in it which is a bash syntax character to run a task in the background. Quote the URL to prevent it being interpreted as syntax.
admin#server:~$ curl -H "Authorization: Bearer <NOT DISPLAYED FOR THIS POST>" -o /home/admin/result.json "https://www.endpoint.com/manage/query/run?id=55408&cmd=service&output=json"
I am new to using API and not quite sure how to use the CoinGlass API to retrieve data.
According to the documentation, it supports using the curl command
curl --location --request GET 'https://open-api.coinglass.com/api/pro/v1/futures/funding_rates_chart?symbol=BTC&type=C' \
--header 'coinglassSecret: fabe11b61f5a4963a4227f4ac2cbcf2f'
https://coinglass.github.io/API-Reference/#liquidation-chart
However, when I tried to run it in the command prompt in Windows 10, it gave me the below error.
C:\Users\User1>curl --request GET 'https://open-api.coinglass.com/api/pro/v1/futures/funding_rates_chart?symbol=BTC&type=C' --header 'coinglassSecret: fabe11b61f5a4963a4227f4ac2cbcf2f'
curl: (3) URL using bad/illegal format or missing URL
Please kindly point me to right direction, million thanks!
I removed the single quotes around the URL. Executed in Win 10 CMD prompt, and got this
curl --request GET https://open-api.coinglass.com/api/pro/v1/futures/funding_rates_chart?symbol=BTC&type=C --header 'coinglassSecret: fabe11b61f5a4963a4227f4ac2cbcf2f'
{"code":"30001","msg":"secret invalid","success":false}
The system cannot find the file specified.
Error occurred while processing: C.
The system cannot find the file specified.
Error occurred while processing: --header.
The filename, directory name, or volume label syntax is incorrect.
I am doing some basic piping of some simple raw code from github to terminal as shown here i.e.
curl https://raw.github.com/leachim6/hello-world/master/r/ruby.rb | ruby
When I try it, it doesn't produce "Hello World", but instead I just see
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
use
curl -sSL https://raw.github.com/leachim6/hello-world/master/r/ruby.rb | ruby
this should work
Update to explain
this URL is redirecting to
https://raw.githubusercontent.com/leachim6/hello-world/master/r/ruby.rb
so -L option was required to follow the redirection (-L, --location)
this option will make curl redo the request on the new place
sS to hide the progress bar and show errors if happened
to debug curl request you can use -v option which will make you see exactly what is happening
Setup
I'm Using curl in the following bash script to push a JSON file to a REST API running in tomcat sitting behind nginx.
while IFS= read -d '' -r file; do
base=$(basename "$file")
datetime=$(find $file -maxdepth 0 -printf "%TY/%Tm/%Td %TH:%TM:%.2TS")
curl -vX POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" \
-d #"$file" -u vangeeij:eian12 \
"http://192.168.105.10/homeaccess/services/aCStats/uploadData?username=vangeeij&filename=$base&datetime=$datetime"
#sudo mv "$file" /home/vangeeij/acserver/resultsOld
done < <(sudo find . -type f -print0)
Problem
When running this script I get a http 400 response with curl error:
curl: (18) transfer closed with outstanding read data remaining
What I have tried
I have found 2 things. First running the same URL and body through Postman yields a successful POST.
I found that this error goes away when the last parameter is removed from the URL &datetime=$datetime
I have also found a few connections between this error and setting a curl option something like
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Expect:'));
But I'm not sure where/how to set this exactly when using curl in a simple bash script
Question
What do I need to change in my curl command to get rid of the error and still be able to use all parameters?
UPDATE
Starting a new question, as further investigation has lead me to a better understanding of the problem.
New Question Link
The error has to do with the fact that the parameter datetime= ends up with text in it that needs to be URL encoded.
This was confirmed by replacing the variable with 2017%2F03%2F01%2008%3A50%3A56
and it worked.
So now the problem is, that I can't get --data-urlencode datetime=$datetime to work. It seems this just gets appended to the JSON data or something.
This error is being generated by the fact that the datetime= paramater is being passed in with non encoded non URL friendly characters... (eg. space).
The fix to this would be to find a way to convert the $datetime to a URLEncoded String.
eg. convert:
2017/03/01 08:50:56
TO
2017%2F03%2F01%2008%3A50%3A56
See the following discussion for one method to accomplish this.
Post JSON data to Rest with URLEncoded query paramaters
I'm learning feathersjs as per below link
http://feathersjs.com/quick-start/
I need to run below command and monitor the output at http://localhost:3000/todo
curl 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
When I tried to run below on cmd, it shows 'curl' is not recognized in cmd prompt.
If I tried to run it using git-bash.exe, bash.exe or sh.exe (in Git/bin or shell.w32-ix86), Cygwin.bat (in cygwin64), it will run fine and showing result in browser.
[{"text":"Do something","id":0}]
But if tried to run it by including into my PATH "C:\Program Files\Git\mingw64\bin, which has curl.exe", I will be getting below error, but "C:\Program Files\Git\usr\bin" will do just fine...
curl: (1) Protocol "'http" not supported or disabled in libcurl
curl: (6) Couldn't resolve host 'application'
curl: (6) Couldn't resolve host text'
curl: (6) Couldn't resolve host 'Do something'
curl: (3) [globbing] unmatched close brace/bracket in column 1
I managed to remove the error with below command
curl "http://localhost:3000/todos/" -H "Content-Type: 'application/json'" --data-binary "{ 'text': 'Do something' }"
But the resulting output will have the Json object "text" missing...
[{"text":"Do something","id":0},{"id":1}]
Question:
1) After modifying the command, the Json object is not parsed successfully. Is there a syntax problem?
2) If there is no syntax problem, does this means that curl need to be run in Unix environment as per original attempt, that it cant be run in cmd directly, but will function ok in bash, cygwin, etc2?
3) What is the difference between curl.exe in C:\Program Files\Git\usr\bin and C:\Program Files\Git\mingw64\bin, which has curl.exe?
Update:
Not OK Raw Cap output
http.content_type == "'application/json'"
OK Raw Cap output
http.content_type == "application/json"
Update2:
Removing single quote in application/json on the 2nd command... shows error
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "Content-Type: 'application/json'" --data-binary "{ 'text': 'Do something' }"
{"id":1}
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "Content-Type: application/json" --data-binary "{ 'text': 'Do something' }"
SyntaxError: Unexpected token '<br> at Object.parse (native)<br> at parse (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\lib\types\json.js:88:17)<br> at C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\lib\read.js:116:18<br> at invokeCallback (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:262:16)<br> at done (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:251:7)<br> at IncomingMessage.onEnd (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:308:7)<br> at emitNone (events.js:67:13)<br> at IncomingMessage.emit (events.js:166:7)<br> at endReadableNT (_stream_readable.js:905:12)<br> at doNTCallback2 (node.js:441:9)
Update 3:
Tried to replace libcurl-4.dll used by curl.exe. Downloaded libcurl from "http://curl.haxx.se/download.html", MingW32 from "http://sourceforge.net/projects/mingw/files/MSYS/" and added "C:\MinGW\bin" to PATH. Then grep -rl "libcurl.dll" . | xargs sed -i 's/libcurl.dll/libcurl-4.dll/g' to create libcurl-4.dll as per suggested "http://curl.haxx.se/mail/lib-2010-11/0174.html". Then execute ./buildconfig, make, make install. Then copied the libcurl-4.dll to C:\Program Files\Git\mingw64\bin folder, but the result is the same...
Update 4
Changing origin of curl.exe but still using same command, shows error on the mingw64 version. I'm suspecting that mingw64 curl need special escaping to make it work?
C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService>"C:\Program Files\Git\mingw64\bin\curl.exe" 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
curl: (1) Protocol "'http" not supported or disabled in libcurl
curl: (6) Couldn't resolve host 'application'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'Do something'
curl: (3) [globbing] unmatched close brace/bracket in column 1
C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService>"C:\Program Files\Git\usr\bin\curl.exe" 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
{"text":"Do something","id":38}
Update 5
From manual... curl --manual
-d, --data <data>
(HTTP) Sends the specified data in a POST request to the HTTP
server, in the same way that a browser does when a user has
filled in an HTML form and presses the submit button. This will
cause curl to pass the data to the server using the content-type
application/x-www-form-urlencoded. Compare to -F, --form.
-d, --data is the same as --data-ascii. --data-raw is almost the
same but does not have a special interpretation of the # charac-
ter. To post data purely binary, you should instead use the
--data-binary option. To URL-encode the value of a form field
you may use --data-urlencode.
If any of these options is used more than once on the same com-
mand line, the data pieces specified will be merged together
with a separating &-symbol. Thus, using '-d name=daniel -d
skill=lousy' would generate a post chunk that looks like
'name=daniel&skill=lousy'.
If you start the data with the letter #, the rest should be a
file name to read the data from, or - if you want curl to read
the data from stdin. Multiple files can also be specified. Post-
ing data from a file named 'foobar' would thus be done with
--data #foobar. When --data is told to read from a file like
that, carriage returns and newlines will be stripped out. If you
don't want the # character to have a special interpretation use
--data-raw instead.
So I tried...
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "'Content-Type:' 'application/json'" --data-binary text=doing --data complete=false
{"text":"doing","complete":"false","id":145}
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "'Content-Type:' 'application/json'" --data-binary text=ding
{"text":"ding","id":146}
But I cant figure out how to make more than 1 word for the JSON object for example instead of "doing", I need "doing something". Seems that MingW64 git curl is accepting different format...
After modifying the curl command it works for you, because you need to use double quote for windows system.
After modifying the command, you have mistakenly added single quote around the application/json. That's why despite of having working curl command server was unsure what you have exactly sent to them!
"Content-Type: 'application/json'"
^ ^ notice the unwanted singles
So it should be
"Content-Type: application/json"
If you do not provide path for any binary (i.e. curl.exe, mysql.exe, php.exe, etc) then system looks for them inside the available paths provided in PATH variable and if they found multiple path there then it will only select one, and I don't know which one!