I am tyring to use curl to upload data. I need to upload the binary data.
Unfortunately i think curl loads the file binary into the memory first so for large files this is an issue. The system I am using has a very small amount of memory available and therefore even a file that is 8mb big is struggling and I get:
curl: option --data-binary: out of memory
curl: try 'curl --help' for more information
I have tried the chunked option, and that doesn't seem to work either.
So far this is what I am trying:
curl --insecure -v --max-time 1200 ... --data-binary #/tmp/sd/record/....mp4 --header Transfer-Encoding: chunked -o UPLOAD_TOKEN -D Media_Binary_Data https://....upload
Is there a way that I can prevent the file from loading the binary into the memory or force it to store this data in an sd card and read it from there?
The reason for the out of memory is that --data and its friends all read the data into memory before sending it off to the server. You can work around that easily by doing -T -X POST, but I still believe you went wrong already in your initial -F test.
From: https://github.com/curl/curl/issues/1385
Related
As I made a batch file to update NirSoft tools, I had a strange experience using wget.
First I downloaded a text file with pad links:
wget http://www.nirsoft.net/pad/pad-links.txt --backups=20 --append-output=C:\Path\Update\LOG\Nirsoft\%Timestamp%_NirSoft.log
After, I used fart-js to delete rows I did not need from the pad-links.txt file. Also I used that program to change the download links to https://www.nirsoft.net/utils, and change the file extensions to .zip.
fart ".\pad-links.txt" "http://www.nirsoft.net/pad" "http://www.nirsoft.net/utils" | tee --append C:\Path\Update\LOG\Nirsoft\%Timestamp%_NirSoft.log
and
fart ".\pad-links.txt" ".xml" ".zip" | tee --append C:\Path\Update\LOG\Nirsoft\%Timestamp%_NirSoft.log
After, to download the programs, I used:
wget --timestamping --input-file=C:\Path\UtilSuit\NirLauncher\Download\pad-links.txt --append-output=C:\Path\Update\LOG\Nirsoft\%Timestamp%_NirSoft.log
Having a look at the log file I found out that not all programs are stored in this location. For example WirelessKeyView is stored in https://www.nirsoft.net/toolsdownload/wirelesskeyview.zip.
Trying to get this file with wget leads to downloaded corrupt files at size of 4kb. The same with cURL and aria2. When I download it with Mozilla, or IDM, I have no problems to get the file. So I tried out wget --auth-no-challenge or wget --header="Accept: text/html" --user-agent="Mozilla/5.0 …"
I also tried cliget, the wget/aria2/curl lines it produced while normal downloading with Mozilla.
wget --header 'Host: www.nirsoft.net' --user-agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' --header 'Accept-Language: de,en-US;q=0.7,en;q=0.3' --referer 'https://www.nirsoft.net/utils/wirelesskeyview.html' --header 'Upgrade-Insecure-Requests: 1' --header 'Sec-Fetch-Dest: document' --header 'Sec-Fetch-Mode: navigate' --header 'Sec-Fetch-Site: same-origin' --header 'Sec-Fetch-User: ?1' --header 'DNT: 1' --header 'Sec-GPC: 1' 'https://www.nirsoft.net/toolsdownload/wirelesskeyview.zip' --output-document 'wirelesskeyview.zip'
I googled and found this reference for powershell, (same error), but cannot reproduce the working answer in batch, (I am not familiar with powershell scripting).
So how is is possible to download the single wirelesskey.zip file with wget/curl or aria2 in a batch script?
A workaround I found out is downloading it directly from the pad Panel but I want the .zip-file, including the updated .chm-file, and also the 64-bit versions, if available.
One more note, within my anti-virus tool the nirsoft site is exempted from scanning, so that is not the answer.
Any solutions?
Aah, this one is simple. If you look at the actual page downloaded, it's called "403.html". So, let's open it. The first thing that strikes you is this:
<title>Error 403: Missing HTTP referer in the HTTP request</title>
So, the server wants a Referer header. Sure, let's give it one:
$ wget --referer foo <URL>
And it downloads the zip file correctly as expected.
Now, really, the server should not be returning a HTTP 200 response with a file called 403. It really should have sent back a HTTP 403 response. But what can you do? There's broken servers everywhere
I've been trying to learn how to use the Google Drive API to update a file in the Google Drive by using a resumable session.
I received a 'Forbidden' response to the upload content request.
Could you help me find missing or misused steps?
User is authorized with permissions:
drive.file (https://www.googleapis.com/auth/drive.file)
Execute a request to create resumable session:
PATCH 'https://www.googleapis.com/upload/drive/v3/files/1XIU63B-U8b9Fe1_UFFVvd7OOdS_ANqAj?uploadType=resumable
Retrieve session url:
https://www.googleapis.com/upload/drive/v3/files/1XIU63B-U8b9Fe1_UFFVvd7OOdS_ANqAj?uploadType=resumable&upload_id=AEnB2Uqew...
Send content by using resumable session:
PUT https://www.googleapis.com/upload/drive/v3/files/1XIU63B-U8b9Fe1_UFFVvd7OOdS_ANqAj?uploadType=resumable&upload_id=AEnB2Uqew...
I didn't find anything specific related to this step in the documentation, so I use regular upload documentation https://developers.google.com/drive/api/v3/manage-uploads#upload-resumable to update file in "Multiple chunks"
I get 403 error status code with 'Forbidden' reason and header with upload_id:
X-GUploader-UploadID: AEnB2Uqewr...
You want to update the existing file in Google Drive with the resumable upload method.
Unfortunately, from your question, I couldn't understand about the detail request body of your test. By this, I cannot replicate your situation. So in this answer, I would like to propose a sample flow for updating the existing file with the resumable upload.
Sample situation:
In this answer, as a sample situation, it supposes that a text file in Google Drive is updated by the resumable upload with the multiple chunks. And as the method for requesting, I use the curl command.
I prepared 2 files for 2 chunks. As the test situation, the 2 chuncs of 262,144 bytes and 37,856 bytes are uploaded. So total upload size is 300,000 bytes.
When you use the resumable upload, please be careful the following point.
Add the chunk's data to the request body. Create chunks in multiples of 256 KB (256 x 1024 bytes) in size, except for the final chunk that completes the upload. Keep the chunk size as large as possible so that the upload is efficient. Ref
Flow for updating a file with the resumable upload:
1. Initiate a resumable upload session
Create the session for uploading with the resumable upload. In this case, the existing file is updated, so the endpoint is PUT https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable. But as an important point, please use the method of PATCH instead of PUT. When PUT is used, location is not included in the response header. I thought that the official document might be not correct.
$ curl -X PATCH -i \
-H "Authorization: Bearer ###accessToken###" \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable"
If you want to update the file as the multipart upload, please use the following sample command. In this case, the filename is changed.
$ curl -X PATCH -i \
-H "Authorization: Bearer ###accessToken###" \
-H "Content-Type: application/json; charset=UTF-8" \
-d '{"name":"updatedFilename.txt"}' \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable"
When above sample command is run, 200 OK is returned, and the response header includes location like location: https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable&upload_id=###. For uploading the data, location is used as the endpoint.
2. Upload the 1st chunk
$ curl -X PUT -i \
-H "Content-Length: 262144" \
-H "Content-Range: bytes 0-262143/300000" \
-H "Content-Type: text/plain" \
-F "file=#data1.txt" \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable&upload_id=###"
When this curl command is run, 308 Resume Incomplete is returned. By this, it is found that the chunk could be correctly uploaded.
3. Upload the 2nd chunk (This is the last chunk of this sample flow.)
$ curl -X PUT -i \
-H "Content-Length: 37856" \
-H "Content-Range: bytes 262144-299999/300000" \
-H "Content-Type: text/plain" \
-F "file=#data2.txt" \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable&upload_id=###"
When this curl command is run, 200 OK is returned, and the file metadata is also returned. By this, it is found that the resumable upload could be correctly done.
Note:
In this case, the file is updated as the overwrite. So please be careful this.
In my environment, even when PUT is modified to PATCH for uploading the chunks, I could confirm that the above flow worked.
If in your environment, an error occurs, please try to test this modification.
About above sample situation, if you want to upload one chunk of 300,000 bytes, please use -H "Content-Length: 300000" -H "Content-Range: bytes 0-299999/300000".
References:
Perform a resumable upload
Setup
I'm Using curl in the following bash script to push a JSON file to a REST API running in tomcat sitting behind nginx.
while IFS= read -d '' -r file; do
base=$(basename "$file")
datetime=$(find $file -maxdepth 0 -printf "%TY/%Tm/%Td %TH:%TM:%.2TS")
curl -vX POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" \
-d #"$file" -u vangeeij:eian12 \
"http://192.168.105.10/homeaccess/services/aCStats/uploadData?username=vangeeij&filename=$base&datetime=$datetime"
#sudo mv "$file" /home/vangeeij/acserver/resultsOld
done < <(sudo find . -type f -print0)
Problem
When running this script I get a http 400 response with curl error:
curl: (18) transfer closed with outstanding read data remaining
What I have tried
I have found 2 things. First running the same URL and body through Postman yields a successful POST.
I found that this error goes away when the last parameter is removed from the URL &datetime=$datetime
I have also found a few connections between this error and setting a curl option something like
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Expect:'));
But I'm not sure where/how to set this exactly when using curl in a simple bash script
Question
What do I need to change in my curl command to get rid of the error and still be able to use all parameters?
UPDATE
Starting a new question, as further investigation has lead me to a better understanding of the problem.
New Question Link
The error has to do with the fact that the parameter datetime= ends up with text in it that needs to be URL encoded.
This was confirmed by replacing the variable with 2017%2F03%2F01%2008%3A50%3A56
and it worked.
So now the problem is, that I can't get --data-urlencode datetime=$datetime to work. It seems this just gets appended to the JSON data or something.
This error is being generated by the fact that the datetime= paramater is being passed in with non encoded non URL friendly characters... (eg. space).
The fix to this would be to find a way to convert the $datetime to a URLEncoded String.
eg. convert:
2017/03/01 08:50:56
TO
2017%2F03%2F01%2008%3A50%3A56
See the following discussion for one method to accomplish this.
Post JSON data to Rest with URLEncoded query paramaters
curl -v -r 0-500 http://somefile -o localfile
It should download just the first 501 bytes, no? Instead, it just downloads the entire thing. All 67 megabytes. Thanks curl! Could my companies proxy servers be blocking this feature somehow? I am skeptical about that, since the downloads themselves do work, just not the range feature. Am I missing something?
As a client you could always abort the download when you have received what you want.
By using head, you will be able to limit the download to 500 bytes, even if the server does not accept the range-header
curl -v -r 0-500 http://somefile |head -c 500 > localfile
It should download just the first 501 bytes, no?
It depends on the server. From man curl:
You should also be aware that many HTTP/1.1 servers do not have this feature enabled, so that when you attempt to get a range, you'll instead get the whole document.
As you can see in the response from the server, it's using HTTP/1.1. So it's not surprising that the range feature is not supported at the server side.
Please use the following command
curl -H "range: bytes=354-500" -O http://example.com/file.extension
How do you POST a binary variable in curl bash?
#!/usr/bin/env bash
IMAGE=$(curl "http://www.google.com/images/srpr/logo3w.png")
curl --data-binary "$IMAGE" --request "POST" "http://www.somesite.com"
Curl seems to do corrupt the image when uploading.
Curl has the option to write response to disk and then read from it, but it'd be more efficient to do it solely in memory.
Try to eliminate the variable ... as follows:
curl "http://www.google.com/images/srpr/logo3w.png" | curl --data-binary - --request "POST" "http://www.somesite.com"
From the curl man page:
If you start the data with the letter #, the rest should be a file name to read the data from, or - if you want curl to read the data from stdin.
EDIT: From the man page, too:
--raw When used, it disables all internal HTTP decoding of content or transfer encodings and instead makes them passed on unaltered, raw. (Added in 7.16.2)
What happens, if applied on either or both sides?
I had a related problem, where I wanted to dynamically curl a file from a given folder.
curl --data-binary directory/$file --request "POST" "http://www.somesite.com"
did not work - uploaded the string "directory/myFile.jar" instead of the actual file.
Adding the # symbol
curl --data-binary #directory/$file --request "POST" "http://www.somesite.com" fixed it.