I've been trying to learn how to use the Google Drive API to update a file in the Google Drive by using a resumable session.
I received a 'Forbidden' response to the upload content request.
Could you help me find missing or misused steps?
User is authorized with permissions:
drive.file (https://www.googleapis.com/auth/drive.file)
Execute a request to create resumable session:
PATCH 'https://www.googleapis.com/upload/drive/v3/files/1XIU63B-U8b9Fe1_UFFVvd7OOdS_ANqAj?uploadType=resumable
Retrieve session url:
https://www.googleapis.com/upload/drive/v3/files/1XIU63B-U8b9Fe1_UFFVvd7OOdS_ANqAj?uploadType=resumable&upload_id=AEnB2Uqew...
Send content by using resumable session:
PUT https://www.googleapis.com/upload/drive/v3/files/1XIU63B-U8b9Fe1_UFFVvd7OOdS_ANqAj?uploadType=resumable&upload_id=AEnB2Uqew...
I didn't find anything specific related to this step in the documentation, so I use regular upload documentation https://developers.google.com/drive/api/v3/manage-uploads#upload-resumable to update file in "Multiple chunks"
I get 403 error status code with 'Forbidden' reason and header with upload_id:
X-GUploader-UploadID: AEnB2Uqewr...
You want to update the existing file in Google Drive with the resumable upload method.
Unfortunately, from your question, I couldn't understand about the detail request body of your test. By this, I cannot replicate your situation. So in this answer, I would like to propose a sample flow for updating the existing file with the resumable upload.
Sample situation:
In this answer, as a sample situation, it supposes that a text file in Google Drive is updated by the resumable upload with the multiple chunks. And as the method for requesting, I use the curl command.
I prepared 2 files for 2 chunks. As the test situation, the 2 chuncs of 262,144 bytes and 37,856 bytes are uploaded. So total upload size is 300,000 bytes.
When you use the resumable upload, please be careful the following point.
Add the chunk's data to the request body. Create chunks in multiples of 256 KB (256 x 1024 bytes) in size, except for the final chunk that completes the upload. Keep the chunk size as large as possible so that the upload is efficient. Ref
Flow for updating a file with the resumable upload:
1. Initiate a resumable upload session
Create the session for uploading with the resumable upload. In this case, the existing file is updated, so the endpoint is PUT https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable. But as an important point, please use the method of PATCH instead of PUT. When PUT is used, location is not included in the response header. I thought that the official document might be not correct.
$ curl -X PATCH -i \
-H "Authorization: Bearer ###accessToken###" \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable"
If you want to update the file as the multipart upload, please use the following sample command. In this case, the filename is changed.
$ curl -X PATCH -i \
-H "Authorization: Bearer ###accessToken###" \
-H "Content-Type: application/json; charset=UTF-8" \
-d '{"name":"updatedFilename.txt"}' \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable"
When above sample command is run, 200 OK is returned, and the response header includes location like location: https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable&upload_id=###. For uploading the data, location is used as the endpoint.
2. Upload the 1st chunk
$ curl -X PUT -i \
-H "Content-Length: 262144" \
-H "Content-Range: bytes 0-262143/300000" \
-H "Content-Type: text/plain" \
-F "file=#data1.txt" \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable&upload_id=###"
When this curl command is run, 308 Resume Incomplete is returned. By this, it is found that the chunk could be correctly uploaded.
3. Upload the 2nd chunk (This is the last chunk of this sample flow.)
$ curl -X PUT -i \
-H "Content-Length: 37856" \
-H "Content-Range: bytes 262144-299999/300000" \
-H "Content-Type: text/plain" \
-F "file=#data2.txt" \
"https://www.googleapis.com/upload/drive/v3/files/[FILE_ID]?uploadType=resumable&upload_id=###"
When this curl command is run, 200 OK is returned, and the file metadata is also returned. By this, it is found that the resumable upload could be correctly done.
Note:
In this case, the file is updated as the overwrite. So please be careful this.
In my environment, even when PUT is modified to PATCH for uploading the chunks, I could confirm that the above flow worked.
If in your environment, an error occurs, please try to test this modification.
About above sample situation, if you want to upload one chunk of 300,000 bytes, please use -H "Content-Length: 300000" -H "Content-Range: bytes 0-299999/300000".
References:
Perform a resumable upload
Related
I'm trying to download my master thesis from google docs as a pdf file..
The ui method (pdf > download > PDF document) fails because of internet connection and the size of the document which is quite big.
I want to download it using wget command from a Linux server which had a good internet connection, I tried a lot of solutions such this one, but they didn't work..
any help please!
If your server can access the user interface of Docs I strongly recommend you to download the PDF with it to save complexity. If the server doesn't offer that option you could use this curl command below to capture Files.export() from Drive API.
curl \
'https://www.googleapis.com/drive/v3/files/{DOC ID}/export?mimeType=application%2Fpdf' \
--header 'Authorization: Bearer {ACCESS TOKEN}' \
--header 'Accept: application/json' \
--compressed >> Doc.pdf
You'll need an access token for a single use, so could use OAuth 2.0 Playground to generate one by selecting the https://www.googleapis.com/auth/drive scope and clicking on Exchange authorization code for tokens.
I wanna make a very simple bash script for downloading files from google drive via Drive API, so in this case there is a big file on google drive and I installed OAuth 2.0 Playground on my google drive account, then in the Select the Scope box, I choose Drive API v3, and https://www.googleapis.com/auth/drive.readonly to make a token and link.
After clicking Authorize APIs and then Exchange authorization code for tokens. I copied the Access tokenlike below.
#! /bin/bash
read -p 'Enter your id : ' id
read -p 'Enter your new token : ' token
read -p 'Enter your file name : ' file
curl -H "Authorization: Bearer $token" "https://www.googleapis.com/drive/v3/files/$id?alt=media" -o "$file"
but it won't work, any idea ?
for example the size of my file is 12G, when I run the code I will get this as output and after a second it back to prompt again ! I checked it in two computers with two different ip addresses.(I also add alt=media to URL)
-bash-3.2# bash mycode.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 166 100 166 0 0 80 0 0:00:02 0:00:02 --:--:-- 80
-bash-3.2#
the content of file that it created is like this
{
"error": {
"errors": [
{
"domain": "global",
"reason": "downloadQuotaExceeded",
"message": "The download quota for this file has been exceeded."
}
],
"code": 403,
"message": "The download quota for this file has been exceeded."
}
}
You want to download a file from Google Drive using the curl command with the access token.
If my understanding is correct, how about this modification?
Modified curl command:
Please add the query parameter of alt=media.
curl -H "Authorization: Bearer $token" "https://www.googleapis.com/drive/v3/files/$id?alt=media" -o "$file"
Note:
This modified curl command supposes that your access token can be used for downloading the file.
In this modification, the files except for Google Docs can be downloaded. If you want to download the Google Docs, please use the Files: export method of Drive API. Ref
Reference:
Download files
If I misunderstood your question and this was not the direction you want, I apologize.
UPDATE AS FOR MARCH 2021
Simply follow this guide here. It worked for me.
In summary:
For small files to download run
wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O FILENAME
While if you are trying to download a quite large file you should try to run
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=FILEID" -O FILENAME && rm -rf /tmp/cookies.txt
Simply substitute FILEID and FILENAME with your custom values.
FILEID can be found in your file share link (after the /d/ as illustrated in the article mantioned above).
FILENAME is simply the name you want to save the download as. Remember to include the right extension. For Example FILENAME = my_file.pdf if the file is a pdf.
This is a known bug
It has been reported in this Issue Tracker post. This is caused because as you can read in the documentation:
(about download url)
Short lived download URL for the file. This field is only populated
for files with content stored in Google Drive; it is not populated for
Google Docs or shortcut files.
So you should use another field.
You can follow the report by clicking on the star next to the issue
number to give more priority to the bug and to receive updates.
As you can read in the comments of the report, the current workaround is:
Use webContentlink instead
or
Change www.googleapis.com to content.googleapis.com
I am tyring to use curl to upload data. I need to upload the binary data.
Unfortunately i think curl loads the file binary into the memory first so for large files this is an issue. The system I am using has a very small amount of memory available and therefore even a file that is 8mb big is struggling and I get:
curl: option --data-binary: out of memory
curl: try 'curl --help' for more information
I have tried the chunked option, and that doesn't seem to work either.
So far this is what I am trying:
curl --insecure -v --max-time 1200 ... --data-binary #/tmp/sd/record/....mp4 --header Transfer-Encoding: chunked -o UPLOAD_TOKEN -D Media_Binary_Data https://....upload
Is there a way that I can prevent the file from loading the binary into the memory or force it to store this data in an sd card and read it from there?
The reason for the out of memory is that --data and its friends all read the data into memory before sending it off to the server. You can work around that easily by doing -T -X POST, but I still believe you went wrong already in your initial -F test.
From: https://github.com/curl/curl/issues/1385
I am trying to upload a zip file to Google drive account using curl.
The file is uploaded successfully but the filename is not getting updated. It gets uploaded with default filename i.e. "Untitled".
I am using below command.
curl -k -H "Authorization: Bearer cat /tmp/token.txt" -F "metadata={name : 'backup.zip'} --data-binary "#backup.zip" https://www.googleapis.com/upload/drive/v2/files?uploadType=multipart
You can use Drive API v3 to upload the zip file. The modified curl code is as follows.
curl -X POST -L \
-H "Authorization: Bearer `cat /tmp/token.txt`" \
-F "metadata={name : 'backup.zip'};type=application/json;charset=UTF-8" \
-F "file=#backup.zip;type=application/zip" \
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart"
In order to use this, please include https://www.googleapis.com/auth/drive in the scope.
The answer above works fine and was the command I used in uploading my file to Google Drive using Curl. However, I didn't understand what scope was and all of the initial setup required to make this command work. Hence, for documentation purposes. I'll give a second answer.
Valid as at the time of writing...
Visit the Credentials page and create a new credential (this is assuming you have created a project). I created credentials for TVs and Limited devices, so the work flow was similar to:
Create credentials > OAuth client ID > Application Type > TVs and Limited Input devices > Named the client > Clicked Create.
After doing this, I was able to copy the Client ID and Client Secret when viewing the newly created credential.
NB: Only the variables with double asterisk from the Curl commands should be replaced.
Next step was to run the Curl command:
curl -d "client_id=**client_id**&scope=**scope**" https://oauth2.googleapis.com/device/code
Scope in this situation can be considered to be the kind of access you intend to have with the credential having the inputted client_id. More about scope from the docs For the use case in focus, which is to upload files, the scope chosen was https://www.googleapis.com/auth/drive.file.
On running the curl command above, you'll get a response similar to:
{ "device_code": "XXXXXXXXXXXXX", "user_code": "ABCD-EFGH",
"expires_in": 1800, "interval": 5, "verification_url":
"https://www.google.com/device" }
Next step is to visit the verification_url in the response in your browser, provide the user_code and accept requests for permissions. You will be presented with a code when all prompts have been followed, this code wasn't required for the remaining steps (but there may be some reasons to use it for other use cases).
Next step is to use the Curl command:
curl -d client_id=**client_id** -d client_secret=**client_secret** -d device_code=**device_code** -d grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Adevice_code https://accounts.google.com/o/oauth2/token
You will get a response similar to:
{ "access_token": "XXXXXXXXX", "expires_in": 3599,
"refresh_token": "XXXXXXXXX", "scope":
"https://www.googleapis.com/auth/drive.file", "token_type": "Bearer"
}
Now you can use the access token and follow the accepted answer with a Curl command similar to:
curl -X POST -L \
-H "Authorization: Bearer **access_token**" \
-F "metadata={name : 'backup.zip'};type=application/json;charset=UTF-8" \
-F "file=#backup.zip;type=application/zip" \
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart"
Setup
I'm Using curl in the following bash script to push a JSON file to a REST API running in tomcat sitting behind nginx.
while IFS= read -d '' -r file; do
base=$(basename "$file")
datetime=$(find $file -maxdepth 0 -printf "%TY/%Tm/%Td %TH:%TM:%.2TS")
curl -vX POST -H "Content-Type: application/json" -H "Cache-Control: no-cache" \
-d #"$file" -u vangeeij:eian12 \
"http://192.168.105.10/homeaccess/services/aCStats/uploadData?username=vangeeij&filename=$base&datetime=$datetime"
#sudo mv "$file" /home/vangeeij/acserver/resultsOld
done < <(sudo find . -type f -print0)
Problem
When running this script I get a http 400 response with curl error:
curl: (18) transfer closed with outstanding read data remaining
What I have tried
I have found 2 things. First running the same URL and body through Postman yields a successful POST.
I found that this error goes away when the last parameter is removed from the URL &datetime=$datetime
I have also found a few connections between this error and setting a curl option something like
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Expect:'));
But I'm not sure where/how to set this exactly when using curl in a simple bash script
Question
What do I need to change in my curl command to get rid of the error and still be able to use all parameters?
UPDATE
Starting a new question, as further investigation has lead me to a better understanding of the problem.
New Question Link
The error has to do with the fact that the parameter datetime= ends up with text in it that needs to be URL encoded.
This was confirmed by replacing the variable with 2017%2F03%2F01%2008%3A50%3A56
and it worked.
So now the problem is, that I can't get --data-urlencode datetime=$datetime to work. It seems this just gets appended to the JSON data or something.
This error is being generated by the fact that the datetime= paramater is being passed in with non encoded non URL friendly characters... (eg. space).
The fix to this would be to find a way to convert the $datetime to a URLEncoded String.
eg. convert:
2017/03/01 08:50:56
TO
2017%2F03%2F01%2008%3A50%3A56
See the following discussion for one method to accomplish this.
Post JSON data to Rest with URLEncoded query paramaters