I want to download data from google drive link shared by someone using google colab. I am a new user of colab and I don't know how to do that.
the links are
x_train: https://drive.google.com/open?id=1cUaIEd9-MLJHFGjLz5QziNvfBtYygplX
y_train: https://drive.google.com/open?id=1hv24Ufiio9rBeSqgnNoM3dr5sVGwOmBy
x_test: https://drive.google.com/open?id=1AH9lKHT5P2oQLz8SGMRPWs_M9wIM2ZRH
y_test: https://drive.google.com/open?id=1i4_azocSDuU3TcDf3OSHO1vF0D5-xMU6
Thanks in advance
You can use gdown if the file is shared publicly.
!gdown 1cUaIEd9-MLJHFGjLz5QziNvfBtYygplX
If it's shared to you only, you need to use pydrive
# Install the PyDrive wrapper & import libraries.
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
# Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
file_id = '1cUaIEd9-MLJHFGjLz5QziNvfBtYygplX'
downloaded = drive.CreateFile({'id':file_id})
downloaded.FetchMetadata(fetch_all=True)
downloaded.GetContentFile(downloaded.metadata['title'])
If they share a folder, it's too long so I made it short in my library.
!pip install kora
from kora import drive
drive.download_folder('1HvIeNhqtFVllXFWzH5NawDdnIfgGDwCK')
A great question and something that I have been working with for some time. The most seamless/ workflow friendly is using gdown.
Presently Colab has a slightly older version install which does not allow full functionality and is installed on pyton2.7 rather than Colab system python. therefore following the terminal/underlining os via `!` method, `!pip installing`:
!pip uninstall gdown -y && pip install gdown
!gdown -V
Then you can use gdown via another ! (method 1) or import gdown (method 2) if you want to use it in code:
Method 1 for whole shared folders/directories:
!gdown --folder https://drive.google.com/drive/folders/sdsldkfj...somefileid.. -O /some_parent_directory/some_child_directory
for files:
!gdown https://drive.google.com/drive/folders/sdsldkfj...somefileid.. -O /some_parent_directory/some_child_directory
Method 2 Using down via importing works as follows:
import gdown
url = 'https://drive.google.com/uc?id=0B9P1L--7Wd2vNm9zMTJWOGxobkU'
output = '20150428_collected_images.tgz'
gdown.download(url, output, quiet=False)
Pasting your google drive file/directory url in both cases.
Hope this helps <^_^>
If you have a link to someone's publicly shared file on Google Drive, you can do this in 4 mouse clicks:
Create a shortcut to this file in your Google Drive (right-click on file -> "Add shortcut to Disk")
Mount your Google drive in Colab notebook (click buttons: "Files" -> "Mount Drive")
Now you can access this file via your shortcut for !unzip/np.load/...
Related
I would like to download an entire public folder from Google-drive from script (python, wget, terminal, etc.).
The procedure shall be accomplished without authentication, as it's a public folder which is accessible for anyone who has the link.
Link example: https://drive.google.com/drive/folders/1Gt-W8jMrADizXYGF5QZwJh_Gc8QpKflX
In case that it's not possible to directly download the entire folder, then it would be sufficient to just being able to list its content (files), without authentication, and then I'll be able to download each file separately. How to obtain such a listing feature?
Note:
I found many similar discussions, but all were assuming either file-download or authentication, and I couldn't find a match for that specific ask, e.g.:
Python: How do download entire folder from Google Drive
download folder from google drive
Download entire Google Drive folder from the shared link using Google drive API
How to download specific Google Drive folder using Python?
Download Shared Google Drive Folder with Python
Python: download files from google drive using url
https://unix.stackexchange.com/questions/136371/how-to-download-a-folder-from-google-drive-using-terminal
https://github.com/vikynandha-zz/google-drive-backup/blob/master/drive.py
I've ended up applying the following code (tested, works well):
import urllib.request
from getfilelistpy import getfilelist
from os import path, makedirs, remove, rename
def download_googledrive_folder(remote_folder, local_dir, gdrive_api_key, debug_en):
success = True
if debug_en:
print('[DEBUG] Downloading: %s --> %s' % (remote_folder, local_dir))
else:
try:
resource = {
"api_key": gdrive_api_key,
"id": remote_folder.split('/')[-1].split('?')[0],
"fields": "files(name,id)",
}
res = getfilelist.GetFileList(resource)
print('Found #%d files' % res['totalNumberOfFiles'])
destination = local_dir
if not path.exists(destination):
makedirs(destination)
for file_dict in res['fileList'][0]['files']:
print('Downloading %s' % file_dict['name'])
if gdrive_api_key:
source = "https://www.googleapis.com/drive/v3/files/%s?alt=media&key=%s" % (file_dict['id'], gdrive_api_key)
else:
source = "https://drive.google.com/uc?id=%s&export=download" % file_dict['id'] # only works for small files (<100MB)
destination_file = path.join(destination, file_dict['name'])
urllib.request.urlretrieve(source, destination_file)
except Exception as err:
print(err)
success = False
return success
I could not find a way to achieve my original goal, i.e. downloading public folder from Google-Drive without any credentials/keys, but the above code is a good compromise for me, as it only requires a key and not full credentials.
Note that there are 2 options here --> with or without providing the Google API Key (gdrive_api_key) for the source URL.
From my experience, the option without the API key works well for small files (< ~100MB), while the option with the API key appear to be more robust and works for any size.
Is there a package in Python3, which can check if a windows directory is shared? and also to share a windows directory?
I am aware of using 'net share' command, I would like to know if there are is a pythonic way of doing it
For check and add, you could use NetShareCheck and NetShareAdd.
And here is an sample to use netapi in python.
Or Use winapi SHGetFileInfo with SHGFI_ATTRIBUTES, then check the dwAttributes flag for SFGAO_SHARE.
You could create it with class Win32_Share and its Create method
import wmi
c = wmi.WMI()
c.Win32_Share().Create(args)
I am trying to make thenao use gpu on windows. This tutorial suggests that I create a .theanorc directory at my home and a theanorc.txt inside it to be able to set the configuration flags before initialization.
Where to create the theanorc.txt file (i.e. how to find out where my home is?) and how to make theano able to see it?
I have tried the following script to create .theanorc and then added theanorc.txt manually inside it, but gpu was not enabled:
import os
_theano_base_dir = os.path.expanduser('~')
if not os.access(_theano_base_dir, os.W_OK):
_theano_base_dir = '/tmp'
_theano_dir = os.path.join(_theano_base_dir, '.theanorc')
if not os.path.exists(_theano_dir):
os.makedirs(_theano_dir)
theano_config_path = os.path.expanduser(os.path.join(_theano_dir, 'theanorc.txt'))
print (theano_config_path)
This printed: C:\SPB_Data\.theanorc\theanorc.txt Is C:\SPB_Data my home?
In Windows, your home directory should be C:\Users\Your_Windows_UserName. Also if you want to create the .theanorc file without the .txt extension you can use Notepad++
I have developed my add-on in scratchpad environment and now developing is finished and I want to create final xpi file.
I replace only this:
Cu.import('resource://gre/modules/ctypes.jsm');
by this:
var {Cu} = require("chrome");
var{ctypes} = Cu.import("resource://gre/modules/ctypes.jsm", null);
Then using nodejs (jpm init and jpm xpi commands) I created xpi file however this is not worked properly.
What we did was follow the jpm tutorial: https://developer.mozilla.org/en-US/Add-ons/SDK/Tutorials/Getting_Started_%28jpm%29 and https://developer.mozilla.org/en-US/Add-ons/SDK/Tools/jpm#Installation
I did this on a Windows system:
we downloaded node.js
npm came with it
created a directory, in this directory i did jpm-init from command line
filled out the prompts then filled in the code for the addon:
We then created a similiar addon to this demo addon here:
https://github.com/Noitidart/jpm-chromeworker
I cant share the actuall addon as that was personal to the user. But the above is simpler and shows how to do it.
We did our jsctypes in a chromeworker, and have it communicate with index.js via messaging
I’m trying to use the dynamic RE panel logit model.
I downloaded the redprob.ado, redprob.hlp, redpmod_ll.ado files from Prof. Mark Stewart's website and installed as follows.
Create ado and personal folder in my C: drive.
Save those 3 files in personal folder.
Typed net set ado c:\ado\personal
Typed adopath + "c:\ado\personal”
Type ssc install redprob
But the following message came out.
ssc install: "redprob" not found at SSC, type -findit redprob-
(To find all packages at SSC that start with r, type -ssc describe r-)”
What is the way to install and use redprob?
As you noted these files should be downloaded from Mark Stewart's site, namely http://www2.warwick.ac.uk/fac/soc/economics/staff/academic/stewart/stata
The third file is called redpmod_ll.ado, and not as you originally typed.
They are not on SSC, so the ssc command is completely irrelevant. What you did is like saying "fetch me this program from Warwick" and then "and now fetch me this program from Connecticut" when it is only at Warwick.
The StataCorp recommendation is to use c:\ado\plus, but what you did should have worked.
Try it with
which redprob
and Stata should be able to find the program and tell you where it is.
If you have manually downloaded the files, simply pasting them into c:\ado\personal should do the work. Stata will automatically load commands from this folder when it starts. Create the folder if it doesn't exist.