I'm implementing an automated, command line file uploader using curl to a servlet.
The problem is, I've got tens of thousands of files with semicolons (;) in the filenames. I'm well aware of the annoyance of this but it is a legacy app that continues to produce new files each day. Renaming is not really an option for compatibility reasons downstream.
I've tried quoting, escaping, converting to "%3b", fully qualifying the path... the obvious stuff... but nothing seems to work, and it fails to send from the client side. I'm on my mac (bundled curl version 7.21.3) but that shouldn't make a difference?
Any ideas?
macbookpro:~$ curl -F upload=#"my file.txt" http://localhost:8080/data/upload
ok
macbookpro:~$ curl -F upload=#"my;file.txt" http://localhost:8080/data/upload
curl: (26) failed creating formpost data
macbookpro:~$ curl -F upload=#"my\;file.txt" http://localhost:8080/data/upload
curl: (26) failed creating formpost data
macbookpro:~$ curl -F upload=#"my\\;file.txt" http://localhost:8080/data/upload
curl: (26) failed creating formpost data
macbookpro:~$
curl uses ; to separate type (or other directives) from the actual name, so I'd simply use stdin instead:
cat 'my;file.txt' | curl -F upload=#- http://localhost:8080/data/upload
You may possibly add filename= directive as well if desired (but without the semicolon in the name!).
According to CURL man pages, when you have ; or , in your form data(file or raw) you should always enclose it in double quotes.
CURL MAN
So just enclose the filename in double quotes and it should work
Related
I wrote a bash script that goes through a list of names finds the appropriate file based on the name and sends the text of the file as an email to the the name's email address. Everything seems to work fine except when the file name includes an umlaut or other such accent.
curl --ssl-reqd --url 'smtps://smtp.gmail.com:465' --user 'me#gmail.com:password' --mail-from 'me#gmail.com' --mail-rcpt 'you#somewhere.com' --upload-file 'thisHasAnUmläut.txt'
The error I get is curl: (56) response reading failed. The file's text itself has nothing special in it.
Question: How should one deal with such accents in this case?
I am using Watson's Speech-To-Text Lite service and I am trying to find a way to automate the loading of new audio files to transcribe. I am very new to Bash and so I'm unclear of even the more rudimentary terms - so I'm finding the problem hard to find a solution for.
For a single use-case, I run the following file (my API key omitted with 'MY APIKEY')
curl -X POST -u "apikey: MY APIKEY" --header "Content-Type: audio/flac" --data-binary "#audiofile_1.flac" "https://gateway-lon.watsonplatform.net/speech-to-text/api/v1/recognize?model=en-US_BroadbandModel&speaker_labels=true" > C:/Users/outputpath/output_1.txt
What I am essentially trying to achieve is to overcome having to manually type and retype the names of the audio files and output. So if I had three (or more) audio files (i.e. audiofile_1, 2, and 3.flac), i would like to create an output file corresponding to each audio file - Some psuedo-code that might help explain what I mean would be
files = [file_1, file_2, file_3]
for file_x in files:
run curl command
save as output_x
You almost got it. You just need to learn some shell syntax:
files=("file_1" "file_2" "file_3")
for file_x in "${files[#]}"
do
curl -X POST -u "apikey: MY APIKEY" --header "Content-Type: audio/flac" --data-binary "#${file_x}" "https://gateway-lon.watsonplatform.net/speech-to-text/api/v1/recognize?model=en-US_BroadbandModel&speaker_labels=true" > "C:/Users/outputpath/${file_x}.txt"
done
First you create the files array with your list of files. Then, you iterate over those files and run the curl command on each of them.
Setup
I'm Using curl in the following bash script to push a JSON file to a REST API running in tomcat sitting behind nginx.
while IFS= read -d '' -r file; do
base=$(basename "$file")
datetime=$(find $file -maxdepth 0 -printf "%TY/%Tm/%Td %TH:%TM:%.2TS")
curl -vX POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" \
-d #"$file" -u vangeeij:eian12 \
"http://192.168.105.10/homeaccess/services/aCStats/uploadData?username=vangeeij&filename=$base&datetime=$datetime"
#sudo mv "$file" /home/vangeeij/acserver/resultsOld
done < <(sudo find . -type f -print0)
Problem
When running this script I get a http 400 response with curl error:
curl: (18) transfer closed with outstanding read data remaining
What I have tried
I have found 2 things. First running the same URL and body through Postman yields a successful POST.
I found that this error goes away when the last parameter is removed from the URL &datetime=$datetime
I have also found a few connections between this error and setting a curl option something like
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Expect:'));
But I'm not sure where/how to set this exactly when using curl in a simple bash script
Question
What do I need to change in my curl command to get rid of the error and still be able to use all parameters?
UPDATE
Starting a new question, as further investigation has lead me to a better understanding of the problem.
New Question Link
The error has to do with the fact that the parameter datetime= ends up with text in it that needs to be URL encoded.
This was confirmed by replacing the variable with 2017%2F03%2F01%2008%3A50%3A56
and it worked.
So now the problem is, that I can't get --data-urlencode datetime=$datetime to work. It seems this just gets appended to the JSON data or something.
This error is being generated by the fact that the datetime= paramater is being passed in with non encoded non URL friendly characters... (eg. space).
The fix to this would be to find a way to convert the $datetime to a URLEncoded String.
eg. convert:
2017/03/01 08:50:56
TO
2017%2F03%2F01%2008%3A50%3A56
See the following discussion for one method to accomplish this.
Post JSON data to Rest with URLEncoded query paramaters
I'm learning feathersjs as per below link
http://feathersjs.com/quick-start/
I need to run below command and monitor the output at http://localhost:3000/todo
curl 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
When I tried to run below on cmd, it shows 'curl' is not recognized in cmd prompt.
If I tried to run it using git-bash.exe, bash.exe or sh.exe (in Git/bin or shell.w32-ix86), Cygwin.bat (in cygwin64), it will run fine and showing result in browser.
[{"text":"Do something","id":0}]
But if tried to run it by including into my PATH "C:\Program Files\Git\mingw64\bin, which has curl.exe", I will be getting below error, but "C:\Program Files\Git\usr\bin" will do just fine...
curl: (1) Protocol "'http" not supported or disabled in libcurl
curl: (6) Couldn't resolve host 'application'
curl: (6) Couldn't resolve host text'
curl: (6) Couldn't resolve host 'Do something'
curl: (3) [globbing] unmatched close brace/bracket in column 1
I managed to remove the error with below command
curl "http://localhost:3000/todos/" -H "Content-Type: 'application/json'" --data-binary "{ 'text': 'Do something' }"
But the resulting output will have the Json object "text" missing...
[{"text":"Do something","id":0},{"id":1}]
Question:
1) After modifying the command, the Json object is not parsed successfully. Is there a syntax problem?
2) If there is no syntax problem, does this means that curl need to be run in Unix environment as per original attempt, that it cant be run in cmd directly, but will function ok in bash, cygwin, etc2?
3) What is the difference between curl.exe in C:\Program Files\Git\usr\bin and C:\Program Files\Git\mingw64\bin, which has curl.exe?
Update:
Not OK Raw Cap output
http.content_type == "'application/json'"
OK Raw Cap output
http.content_type == "application/json"
Update2:
Removing single quote in application/json on the 2nd command... shows error
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "Content-Type: 'application/json'" --data-binary "{ 'text': 'Do something' }"
{"id":1}
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "Content-Type: application/json" --data-binary "{ 'text': 'Do something' }"
SyntaxError: Unexpected token '<br> at Object.parse (native)<br> at parse (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\lib\types\json.js:88:17)<br> at C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\lib\read.js:116:18<br> at invokeCallback (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:262:16)<br> at done (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:251:7)<br> at IncomingMessage.onEnd (C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService\node_modules\body-parser\node_modules\raw-body\index.js:308:7)<br> at emitNone (events.js:67:13)<br> at IncomingMessage.emit (events.js:166:7)<br> at endReadableNT (_stream_readable.js:905:12)<br> at doNTCallback2 (node.js:441:9)
Update 3:
Tried to replace libcurl-4.dll used by curl.exe. Downloaded libcurl from "http://curl.haxx.se/download.html", MingW32 from "http://sourceforge.net/projects/mingw/files/MSYS/" and added "C:\MinGW\bin" to PATH. Then grep -rl "libcurl.dll" . | xargs sed -i 's/libcurl.dll/libcurl-4.dll/g' to create libcurl-4.dll as per suggested "http://curl.haxx.se/mail/lib-2010-11/0174.html". Then execute ./buildconfig, make, make install. Then copied the libcurl-4.dll to C:\Program Files\Git\mingw64\bin folder, but the result is the same...
Update 4
Changing origin of curl.exe but still using same command, shows error on the mingw64 version. I'm suspecting that mingw64 curl need special escaping to make it work?
C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService>"C:\Program Files\Git\mingw64\bin\curl.exe" 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
curl: (1) Protocol "'http" not supported or disabled in libcurl
curl: (6) Couldn't resolve host 'application'
curl: (6) Couldn't resolve host 'text'
curl: (6) Couldn't resolve host 'Do something'
curl: (3) [globbing] unmatched close brace/bracket in column 1
C:\Users\testUser\Documents\Framework\Javascript\featherstestNewService>"C:\Program Files\Git\usr\bin\curl.exe" 'http://localhost:3000/todos/' -H 'Content-Type: application/json' --data-binary '{ "text": "Do something" }'
{"text":"Do something","id":38}
Update 5
From manual... curl --manual
-d, --data <data>
(HTTP) Sends the specified data in a POST request to the HTTP
server, in the same way that a browser does when a user has
filled in an HTML form and presses the submit button. This will
cause curl to pass the data to the server using the content-type
application/x-www-form-urlencoded. Compare to -F, --form.
-d, --data is the same as --data-ascii. --data-raw is almost the
same but does not have a special interpretation of the # charac-
ter. To post data purely binary, you should instead use the
--data-binary option. To URL-encode the value of a form field
you may use --data-urlencode.
If any of these options is used more than once on the same com-
mand line, the data pieces specified will be merged together
with a separating &-symbol. Thus, using '-d name=daniel -d
skill=lousy' would generate a post chunk that looks like
'name=daniel&skill=lousy'.
If you start the data with the letter #, the rest should be a
file name to read the data from, or - if you want curl to read
the data from stdin. Multiple files can also be specified. Post-
ing data from a file named 'foobar' would thus be done with
--data #foobar. When --data is told to read from a file like
that, carriage returns and newlines will be stripped out. If you
don't want the # character to have a special interpretation use
--data-raw instead.
So I tried...
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "'Content-Type:' 'application/json'" --data-binary text=doing --data complete=false
{"text":"doing","complete":"false","id":145}
C:\Users\testUser>"C:\Program Files\Git\mingw64\bin\curl.exe" "http://localhost:3000/todos/" -H "'Content-Type:' 'application/json'" --data-binary text=ding
{"text":"ding","id":146}
But I cant figure out how to make more than 1 word for the JSON object for example instead of "doing", I need "doing something". Seems that MingW64 git curl is accepting different format...
After modifying the curl command it works for you, because you need to use double quote for windows system.
After modifying the command, you have mistakenly added single quote around the application/json. That's why despite of having working curl command server was unsure what you have exactly sent to them!
"Content-Type: 'application/json'"
^ ^ notice the unwanted singles
So it should be
"Content-Type: application/json"
If you do not provide path for any binary (i.e. curl.exe, mysql.exe, php.exe, etc) then system looks for them inside the available paths provided in PATH variable and if they found multiple path there then it will only select one, and I don't know which one!
I'm writing a Bash script to download image files from Snapito's web page snapshot API. The API can return a variety of responses indicated by different HTTP response codes and/or some custom headers. My script is intended to be run as an automated Cron job that pulls URLs from a MySQL database and saves the screenshots to local disk.
I am using curl. I'd like to do these 3 things using a single CURL command:
Extract the HTTP response code
Extract the headers
Save the file locally (if the request was successful)
I could do this using multiple curl requests, but I want to minimize the number of times I hit Snapito's servers. Any curl experts out there?
Or if someone has a Bash script that can respond to the full documented set of Snapito API responses, that'd be awesome. Here's their API documentation.
Thanks!
Use the dump headers option:
curl -D /tmp/headers.txt http://server.com
Use curl -i (include HTTP header) - which will yield the headers, followed by a blank line, followed by the content.
You can then split out the headers / content (or use -D to save directly to file, as suggested above).
There are three options -i, -I, and -D
> curl --help | egrep '^ +\-[iID]'
-D, --dump-header FILE Write the headers to FILE
-I, --head Show document info only
-i, --include Include protocol headers in the output (H/F)