curl error 18 transfer closed with outstanding read data remaining - bash

Setup
I'm Using curl in the following bash script to push a JSON file to a REST API running in tomcat sitting behind nginx.
while IFS= read -d '' -r file; do
base=$(basename "$file")
datetime=$(find $file -maxdepth 0 -printf "%TY/%Tm/%Td %TH:%TM:%.2TS")
curl -vX POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" \
-d #"$file" -u vangeeij:eian12 \
"http://192.168.105.10/homeaccess/services/aCStats/uploadData?username=vangeeij&filename=$base&datetime=$datetime"
#sudo mv "$file" /home/vangeeij/acserver/resultsOld
done < <(sudo find . -type f -print0)
Problem
When running this script I get a http 400 response with curl error:
curl: (18) transfer closed with outstanding read data remaining
What I have tried
I have found 2 things. First running the same URL and body through Postman yields a successful POST.
I found that this error goes away when the last parameter is removed from the URL &datetime=$datetime
I have also found a few connections between this error and setting a curl option something like
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Expect:'));
But I'm not sure where/how to set this exactly when using curl in a simple bash script
Question
What do I need to change in my curl command to get rid of the error and still be able to use all parameters?
UPDATE
Starting a new question, as further investigation has lead me to a better understanding of the problem.
New Question Link
The error has to do with the fact that the parameter datetime= ends up with text in it that needs to be URL encoded.
This was confirmed by replacing the variable with 2017%2F03%2F01%2008%3A50%3A56
and it worked.
So now the problem is, that I can't get --data-urlencode datetime=$datetime to work. It seems this just gets appended to the JSON data or something.

This error is being generated by the fact that the datetime= paramater is being passed in with non encoded non URL friendly characters... (eg. space).
The fix to this would be to find a way to convert the $datetime to a URLEncoded String.
eg. convert:
2017/03/01 08:50:56
TO
2017%2F03%2F01%2008%3A50%3A56
See the following discussion for one method to accomplish this.
Post JSON data to Rest with URLEncoded query paramaters

Related

Unable to update file in the Google Drive by using resumable approach

I've been trying to learn how to use the Google Drive API to update a file in the Google Drive by using a resumable session.
I received a 'Forbidden' response to the upload content request.
Could you help me find missing or misused steps?
User is authorized with permissions:
drive.file (https://www.googleapis.com/auth/drive.file)
Execute a request to create resumable session:
PATCH 'https://www.googleapis.com/upload/drive/v3/files/1XIU63B-U8b9Fe1_UFFVvd7OOdS_ANqAj?uploadType=resumable
Retrieve session url:
https://www.googleapis.com/upload/drive/v3/files/1XIU63B-U8b9Fe1_UFFVvd7OOdS_ANqAj?uploadType=resumable&upload_id=AEnB2Uqew...
Send content by using resumable session:
PUT https://www.googleapis.com/upload/drive/v3/files/1XIU63B-U8b9Fe1_UFFVvd7OOdS_ANqAj?uploadType=resumable&upload_id=AEnB2Uqew...
I didn't find anything specific related to this step in the documentation, so I use regular upload documentation https://developers.google.com/drive/api/v3/manage-uploads#upload-resumable to update file in "Multiple chunks"
I get 403 error status code with 'Forbidden' reason and header with upload_id:
X-GUploader-UploadID: AEnB2Uqewr...
You want to update the existing file in Google Drive with the resumable upload method.
Unfortunately, from your question, I couldn't understand about the detail request body of your test. By this, I cannot replicate your situation. So in this answer, I would like to propose a sample flow for updating the existing file with the resumable upload.
Sample situation:
In this answer, as a sample situation, it supposes that a text file in Google Drive is updated by the resumable upload with the multiple chunks. And as the method for requesting, I use the curl command.
I prepared 2 files for 2 chunks. As the test situation, the 2 chuncs of 262,144 bytes and 37,856 bytes are uploaded. So total upload size is 300,000 bytes.
When you use the resumable upload, please be careful the following point.
Add the chunk's data to the request body. Create chunks in multiples of 256 KB (256 x 1024 bytes) in size, except for the final chunk that completes the upload. Keep the chunk size as large as possible so that the upload is efficient. Ref
Flow for updating a file with the resumable upload:
1. Initiate a resumable upload session
Create the session for uploading with the resumable upload. In this case, the existing file is updated, so the endpoint is PUT https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable. But as an important point, please use the method of PATCH instead of PUT. When PUT is used, location is not included in the response header. I thought that the official document might be not correct.
$ curl -X PATCH -i \
-H "Authorization: Bearer ###accessToken###" \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable"
If you want to update the file as the multipart upload, please use the following sample command. In this case, the filename is changed.
$ curl -X PATCH -i \
-H "Authorization: Bearer ###accessToken###" \
-H "Content-Type: application/json; charset=UTF-8" \
-d '{"name":"updatedFilename.txt"}' \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable"
When above sample command is run, 200 OK is returned, and the response header includes location like location: https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable&upload_id=###. For uploading the data, location is used as the endpoint.
2. Upload the 1st chunk
$ curl -X PUT -i \
-H "Content-Length: 262144" \
-H "Content-Range: bytes 0-262143/300000" \
-H "Content-Type: text/plain" \
-F "file=#data1.txt" \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable&upload_id=###"
When this curl command is run, 308 Resume Incomplete is returned. By this, it is found that the chunk could be correctly uploaded.
3. Upload the 2nd chunk (This is the last chunk of this sample flow.)
$ curl -X PUT -i \
-H "Content-Length: 37856" \
-H "Content-Range: bytes 262144-299999/300000" \
-H "Content-Type: text/plain" \
-F "file=#data2.txt" \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable&upload_id=###"
When this curl command is run, 200 OK is returned, and the file metadata is also returned. By this, it is found that the resumable upload could be correctly done.
Note:
In this case, the file is updated as the overwrite. So please be careful this.
In my environment, even when PUT is modified to PATCH for uploading the chunks, I could confirm that the above flow worked.
If in your environment, an error occurs, please try to test this modification.
About above sample situation, if you want to upload one chunk of 300,000 bytes, please use -H "Content-Length: 300000" -H "Content-Range: bytes 0-299999/300000".
References:
Perform a resumable upload

Load GeoJSON file into Apache CouchDB

I am working on Windows10 and tried to load a geojson file into my couchdb via the "curl" command and a POST request in the cmd which looks like that:
C:\Program Files\cURL\bin>curl -d #path-to-my-data\data.geojson -H "Content-type: application/json" -X POST http://127.0.0.1:5984/_utils/database.html?-dbName-
and then I get the following error:
{"error":"method_not_allowed","reason":"Only GET,HEAD allowed"}
On http://couchdb-13.readthedocs.org/en/latest/api-basics/ it is said, that "If you use the an unsupported HTTP request type with a URL that does not support the specified type, a 405 error will be returned, listing the supported HTTP methods."
When I try that with a PUT request, I get the same error.
I validated the json with jsonlint so this should not be the problem.
I tried several tutorials like "Three Steps to CouchDB Heaven …" or "Export & Import a Database with CouchDB" but none of them seems to work.
So I am not sure, where the problem is. Do I need to make changes in my geojson file, or something else?
thanks for your help
The needed curl command just looks like that:
curl -H "Content-Type: application/json" -X POST http://localhost:5984/db -d #C:\Users\Name\Desktop\data.geojson

Curl command in Windows

I'm learning feathersjs as per below link
http://feathersjs.com/quick-start/
I need to run below command and monitor the output at http://localhost:3000/todo
curl 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
When I tried to run below on cmd, it shows 'curl' is not recognized in cmd prompt.
If I tried to run it using git-bash.exe, bash.exe or sh.exe (in Git/bin or shell.w32-ix86), Cygwin.bat (in cygwin64), it will run fine and showing result in browser.
[{"text":"Do something","id":0}]
But if tried to run it by including into my PATH "C:\Program Files\Git\mingw64\bin, which has curl.exe", I will be getting below error, but "C:\Program Files\Git\usr\bin" will do just fine...
curl: (1) Protocol "'http" not supported or disabled in libcurl
curl: (6) Couldn't resolve host 'application'
curl: (6) Couldn't resolve host text'
curl: (6) Couldn't resolve host 'Do something'
curl: (3) [globbing] unmatched close brace/bracket in column 1
I managed to remove the error with below command
curl "http://localhost:3000/todos/" -H "Content-Type: 'application/json'" --data-binary "{ 'text': 'Do something' }"
But the resulting output will have the Json object "text" missing...
[{"text":"Do something","id":0},{"id":1}]
Question:
1) After modifying the command, the Json object is not parsed successfully. Is there a syntax problem?
2) If there is no syntax problem, does this means that curl need to be run in Unix environment as per original attempt, that it cant be run in cmd directly, but will function ok in bash, cygwin, etc2?
3) What is the difference between curl.exe in C:\Program Files\Git\usr\bin and C:\Program Files\Git\mingw64\bin, which has curl.exe?
Update:
Not OK Raw Cap output
http.content_type == "'application/json'"
OK Raw Cap output
http.content_type == "application/json"
Update2:
Removing single quote in application/json on the 2nd command... shows error
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "Content-Type: 'application/json'" --data-binary "{ 'text': 'Do something' }"
{"id":1}
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "Content-Type: application/json" --data-binary "{ 'text': 'Do something' }"
SyntaxError: Unexpected token '<br> at Object.parse (native)<br> at parse (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\lib\types\json.js:88:17)<br> at C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\lib\read.js:116:18<br> at invokeCallback (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:262:16)<br> at done (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:251:7)<br> at IncomingMessage.onEnd (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:308:7)<br> at emitNone (events.js:67:13)<br> at IncomingMessage.emit (events.js:166:7)<br> at endReadableNT (_stream_readable.js:905:12)<br> at doNTCallback2 (node.js:441:9)
Update 3:
Tried to replace libcurl-4.dll used by curl.exe. Downloaded libcurl from "http://curl.haxx.se/download.html", MingW32 from "http://sourceforge.net/projects/mingw/files/MSYS/" and added "C:\MinGW\bin" to PATH. Then grep -rl "libcurl.dll" . | xargs sed -i 's/libcurl.dll/libcurl-4.dll/g' to create libcurl-4.dll as per suggested "http://curl.haxx.se/mail/lib-2010-11/0174.html". Then execute ./buildconfig, make, make install. Then copied the libcurl-4.dll to C:\Program Files\Git\mingw64\bin folder, but the result is the same...
Update 4
Changing origin of curl.exe but still using same command, shows error on the mingw64 version. I'm suspecting that mingw64 curl need special escaping to make it work?
C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService>"C:\Program Files\Git\mingw64\bin\curl.exe" 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
curl: (1) Protocol "'http" not supported or disabled in libcurl
curl: (6) Couldn't resolve host 'application'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'Do something'
curl: (3) [globbing] unmatched close brace/bracket in column 1
C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService>"C:\Program Files\Git\usr\bin\curl.exe" 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
{"text":"Do something","id":38}
Update 5
From manual... curl --manual
-d, --data <data>
(HTTP) Sends the specified data in a POST request to the HTTP
server, in the same way that a browser does when a user has
filled in an HTML form and presses the submit button. This will
cause curl to pass the data to the server using the content-type
application/x-www-form-urlencoded. Compare to -F, --form.
-d, --data is the same as --data-ascii. --data-raw is almost the
same but does not have a special interpretation of the # charac-
ter. To post data purely binary, you should instead use the
--data-binary option. To URL-encode the value of a form field
you may use --data-urlencode.
If any of these options is used more than once on the same com-
mand line, the data pieces specified will be merged together
with a separating &-symbol. Thus, using '-d name=daniel -d
skill=lousy' would generate a post chunk that looks like
'name=daniel&skill=lousy'.
If you start the data with the letter #, the rest should be a
file name to read the data from, or - if you want curl to read
the data from stdin. Multiple files can also be specified. Post-
ing data from a file named 'foobar' would thus be done with
--data #foobar. When --data is told to read from a file like
that, carriage returns and newlines will be stripped out. If you
don't want the # character to have a special interpretation use
--data-raw instead.
So I tried...
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "'Content-Type:' 'application/json'" --data-binary text=doing --data complete=false
{"text":"doing","complete":"false","id":145}
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "'Content-Type:' 'application/json'" --data-binary text=ding
{"text":"ding","id":146}
But I cant figure out how to make more than 1 word for the JSON object for example instead of "doing", I need "doing something". Seems that MingW64 git curl is accepting different format...
After modifying the curl command it works for you, because you need to use double quote for windows system.
After modifying the command, you have mistakenly added single quote around the application/json. That's why despite of having working curl command server was unsure what you have exactly sent to them!
"Content-Type: 'application/json'"
^ ^ notice the unwanted singles
So it should be
"Content-Type: application/json"
If you do not provide path for any binary (i.e. curl.exe, mysql.exe, php.exe, etc) then system looks for them inside the available paths provided in PATH variable and if they found multiple path there then it will only select one, and I don't know which one!

Bash curl POST a binary variable

How do you POST a binary variable in curl bash?
#!/usr/bin/env bash
IMAGE=$(curl "http://www.google.com/images/srpr/logo3w.png")
curl --data-binary "$IMAGE" --request "POST" "http://www.somesite.com"
Curl seems to do corrupt the image when uploading.
Curl has the option to write response to disk and then read from it, but it'd be more efficient to do it solely in memory.
Try to eliminate the variable ... as follows:
curl "http://www.google.com/images/srpr/logo3w.png" | curl --data-binary - --request "POST" "http://www.somesite.com"
From the curl man page:
If you start the data with the letter #, the rest should be a file name to read the data from, or - if you want curl to read the data from stdin.
EDIT: From the man page, too:
--raw When used, it disables all internal HTTP decoding of content or transfer encodings and instead makes them passed on unaltered, raw. (Added in 7.16.2)
What happens, if applied on either or both sides?
I had a related problem, where I wanted to dynamically curl a file from a given folder.
curl --data-binary directory/$file --request "POST" "http://www.somesite.com"
did not work - uploaded the string "directory/myFile.jar" instead of the actual file.
Adding the # symbol
curl --data-binary #directory/$file --request "POST" "http://www.somesite.com" fixed it.

curl file upload with semicolons in filename

I'm implementing an automated, command line file uploader using curl to a servlet.
The problem is, I've got tens of thousands of files with semicolons (;) in the filenames. I'm well aware of the annoyance of this but it is a legacy app that continues to produce new files each day. Renaming is not really an option for compatibility reasons downstream.
I've tried quoting, escaping, converting to "%3b", fully qualifying the path... the obvious stuff... but nothing seems to work, and it fails to send from the client side. I'm on my mac (bundled curl version 7.21.3) but that shouldn't make a difference?
Any ideas?
macbookpro:~$ curl -F upload=#"my file.txt" http://localhost:8080/data/upload
ok
macbookpro:~$ curl -F upload=#"my;file.txt" http://localhost:8080/data/upload
curl: (26) failed creating formpost data
macbookpro:~$ curl -F upload=#"my\;file.txt" http://localhost:8080/data/upload
curl: (26) failed creating formpost data
macbookpro:~$ curl -F upload=#"my\\;file.txt" http://localhost:8080/data/upload
curl: (26) failed creating formpost data
macbookpro:~$
curl uses ; to separate type (or other directives) from the actual name, so I'd simply use stdin instead:
cat 'my;file.txt' | curl -F upload=#- http://localhost:8080/data/upload
You may possibly add filename= directive as well if desired (but without the semicolon in the name!).
According to CURL man pages, when you have ; or , in your form data(file or raw) you should always enclose it in double quotes.
CURL MAN
So just enclose the filename in double quotes and it should work

Resources