I'm trying to get a basic app running with Flask + PyTorch, and host it on Heroku. However, I run into the issue that the maximum slug size is 500mb on the free version, and PyTorch itself is ~500mb.
After some google searching, someone wrote about finding a cpu-only version of PyTorch, and using that, which is much smaller here.
However, I'm pretty lost as to how this is done, and the person didn't document this at all. Any advice is appreciated, thanks.
EDIT:
To be more specific about my problem, I tried installing torch by (as far as I understand), including a requirements.txt which listed torch as a dependency. Current I have: torch==0.4.1. However this doesn't work bc of size.
My question is, do you know what I could write in the requirements file to get the cpu-only version of torch that is smaller, or alternatively, if the requirements.txt doesn't work for this, what I would do instead, to get the cpu version.
Per the Pytorch website, you can install pytorch-cpu with
conda install pytorch-cpu torchvision-cpu -c pytorch
You can see from the files on Anaconda cloud, that the size varies between 26 and 56MB depending on the OS where you want to install it.
You can get the wheel from http://download.pytorch.org/whl/cpu/.
The wheel is 87MB.
You can setup the installation by putting the link to the wheel in the requirements.txt file. If you use Python 3.6 on Heroku:
http://download.pytorch.org/whl/cpu/torch-0.4.1-cp36-cp36m-linux_x86_64.whl
otherwise, for Python 2.7:
http://download.pytorch.org/whl/cpu/torch-0.4.1-cp27-cp27mu-linux_x86_64.whl
For example if your requirements are pytorch-cpu, numpy and scipy and you're using Python 3.6, the requirements.txt would look like:
http://download.pytorch.org/whl/cpu/torch-0.4.1-cp36-cp36m-linux_x86_64.whl
numpy
scipy
As of PyTorch 1.3, PyTorch has changed its API. In order to install CPU version only, use
conda install pytorch torchvision cpuonly -c pytorch
And, the corresponding wheel files can be downloaded from https://download.pytorch.org/whl/torch_stable.html and can be installed using pip or use the command similar to the following corresponding to your intended pytorch and torchvision versions
On Linux:
pip3 install torch==1.9.0+cpu torchvision==0.10.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
On Windows / Mac:
pip3 install torch torchvision
Check the PyTorch's getting started guide.
In 2020, please use the following command if you want to download pytorch-cpu version with pip3 (on Linux and Win):
pip3 install torch==1.5.0+cpu torchvision==0.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
I'm getting errors for each version from list of torch stable versions. like
`{specific_version} is not a supported wheel on this platform
Try to put this into your requirements.txt
// requirements.txt
-f https://download.pytorch.org/whl/torch_stable.html
torch==1.8.1+cpu
torchvision==0.9.1+cpu
fastai>=2.3.1
ipywidgets
voila
If you want to install stable pytorch=1.4.0 cpu version using requirements.txt then specify direct download http link...
So that pip will download and install directly...
http://download.pytorch.org/whl/cpu/torch-1.4.0%2Bcpu-cp37-cp37m-linux_x86_64.whl
Alternatively if using terminal or cmd
torch==1.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
For more versions, visit
https://download.pytorch.org/whl/torch_stable.html and choose version as per your requirement (windows, linux, mac version all can be seen in the link)
The problem is the size of the libs, when you use an application locally you can use the GPU resources, as you will not use this on the server, use the following code in requirements.txt:
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==1.11.0+cpu
--find-links https://download.pytorch.org/whl/torch_stable.html
torchvision==0.12.0+cpu
You can use pip to download the latest CPU-only pytorch wheel directly from the pytorch.org website:
pip install torch --extra-index-url https://download.pytorch.org/whl/cpu
Coming to this question after running into the same issue with Heroku's App platform -- slug size well over the 500MB limit. The current instruction from the official PyTorch "Getting Started" page is as follows:
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu
That's for a Linux install, using the latest (1.13.1) stable version, in Python, with pip. Note: the syntax varies based on the system, package manager, language, and preferred build. (See below)
The correct answer is to search for it on the Pytorch website, here: https://pytorch.org/get-started/previous-versions/
They have a complete list of all of the previous versions, as well as the pip syntax to use for each one (including CPU-only versions).
Related
Could anyone if used PyTorch geometric before, help me resolve this issue. I'm having trouble installing torch-scatter from PyTorch Geometric to deal with some tabular data for question answering task based on TAPAS model. I presume there is a compile error at source. I tried checking other forums and found no solution for this yet.
Procedure followed to produce the error:
pip3 install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
pip3 install torch-scatter
Console output:
ERROR: Command errored out with exit status 1:
I also tried using the python -f flag and specifically tried to pull from the source at:
pip3 install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.1+cpu.html
Following are my PyTorch and CUDA versions with the respective imports and console outputs:
python -c "import torch; print(torch.__version__)"
Output:
1.8.1+cpu
CUDA version:
python -c "import torch; print(torch.version.cuda)"
Output:
None
Python version:
Python 3.7.5
Thank you very much for your time and guidance.
Solution found:
The error is due to wrong/unsupported GCC versions (that's indeed more PyTorch related). I tried installing from binaries which couldn't find compatible version. After some research, I found that Pytorch geometric installation documentation suggests that only PyTorch versions 1.4.0, 1.5.0, 1.6.0, 1.7.0, and 1.8.0 are compatible, but does not explicitly mentions this.
Following solution worked:
Downgrading PyTorch to any of the above versions solved the issue. Make sure that you use the same version of PyTorch and torch-scatter while installation. In my case, 1.8.0
I had this problem too which is solved by install C++ build tools. You can install it from vs_buildtools.exe that is downloadable here
I also had some issues related to the torch_geometric, and I resolved those issues after some research. Following are the solutions that I have applied:
After installing torch, please install the following libs:
1. torch-scatter
2. torch-sparse
3. torch-cluster
Then install torch-geometric.
You can install the above packages using pip command (pip install ), If it does not work install through HTML link by selecting your specific version.
Command: pip install libname -f link
Example: pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.12.0%2Bcpu.html
For my case, I have used the torch 1.12 and install the above packages from the above link.
Thanks
I am working with Anaconda with python 2.7. In order to do algorithmic trading I wanted to install 'zipline' package using conda giving command as
conda install -c Quantopian Zipline
from Anaconda prompt. After 'Solving environment' message, I got 'Package Plan' which contains packages which will be installed, removed, updated and downgraded. I was astonished to see that it will remove 'anaconda: 5.2.0-py27_3' and downgrade
networkx: 2.1-py27_0 to 1.11-py27_1;
numpy: 1.14.3-py27h911edcf_1 to 1.11.3-py27hc42714f_10;
numpy-base: 1.14.3-py27h917549b_1 to 1.11.3-py27h2753ae9_10;
pandas: 0.23.0-py27h39f3610_0 to 0.22.0-py27hc56fc5f_0.
I canceled the installation.
I have a couple of question here.
Why at all it is necessary for any package installation to remove package 'Anaconda' and downgrade packages like 'numpy', 'pandas' etc.?
Will this action not jeopardize my other python activities?
Shall I go ahead or restrain from installing the packages like this?
Zipline doesn't currently support the latest versions of packages like panda, numpy etc. which causes the messages above.
Well, yes it could make trouble, especially if your other python activities need the latest version of those packages.
Please don't go ahead with the installation like this. I'll explain the best available solution below.
Solution:
Create an environment for Zipline. Let's say (for convenience only) Zipline supports Python 3.5 but you have only installed Python 2.7 on your machine.
So you can create a sandbox-like conda-environment for Python 3.5. It's very straight forward, just use the following commands:
$ conda create -n env_zipline python=3.5
After your isolated environment called env_zipline was created, you have to activated it by using the following command:
$ activate env_zipline
You can install Zipline now by running
(env_zipline)$ conda install -c Quantopian zipline
When you finished your work with zipline you can deactivate the environment for zipline by using the following command:
(env_zipline)$ deactivate
Hope it helps. If your need further information you can check the more detailed documentation of zipline (the steps above are included):
http://www.zipline.io/install.html
after following the instruction in http://docs.h2o.ai/h2o/latest-stable/h2o-docs/downloading.html#install-in-python, I was able to install h2o v 3.16.0.2. I was also able to use command line instructions mentioned in the above website and test that it is working.
python
import h2o
h2o.init()
h2o.demo("glm")
However, when I launch anaconda spyder, I am not able to import h2o. How do I link the h2o I have installed and bring it into Spyder python?
Update:
I have already tried {conda install -c anaconda h2o } which is mentioned on Anaconda.org, but that installs older ver 3.10 of h2o and that did not work either.
thanks for your help.
I believe the problem is due to your python environment. When you install Anaconda, you need to use anaconda pip, to ensure that the installed packages are available in conda.
The short answer is you can install the lastest stable version of H2O (3.16.0.2 as of today) using conda via h2oai channel instead of anaconda channel (h2o maintains it's own channel):
conda install -c h2oai h2o
this should solve your issue.
But more generally, the packages will appear in conda if you use anaconda pip. You can check which pip is being used by doing
which pip
and making sure that the path to the pip is within your anaconda distribution; something like /home/<userdir>/anaconda/bin/pip instead of /usr/bin/pip
Same is true also for Python. Try checking if starting Python in terminal points to anaconda Python by doing which python. If that's not the case, than you would need to add the conda installation of Python to your PATH variable. Please refer to conda docs for instructions https://conda.io/docs/user-guide/install/index.html
It would have been helpful if you had included information regarding your operating system in the question.
I am interested in getting TensorFlow to run on Windows, however at present I realize that this is not possible due to some of the dependencies not being usable with Windows, e.g. Bazel.
The need arises because as I currently understand it the only way to access the GPU from TensorFlow is via a non-virtual install of Linux. I do realize I can dual boot into a Linux install, but would prefer to avoid that route.
To resolve the problem I am in need of the entire dependency chain to build TensorFlow as was wondering if this already existed.
I also realize that I can capture the build output when building from source as a solid start, but would like to avoid that work if it is already known.
There is a beta of Bazel that runs on Windows - https://github.com/dslomov/bazel-windows
See related GitHub Issue to run TensorFlow on Windows. - https://github.com/tensorflow/tensorflow/issues/17
Another reason to run on Windows is the possibility to port to Xbox One.
I found a possible answer, still need to check it. This will generate a dependency graph as a dot file.
$ bazel query 'deps(//tensorflow/tools/pip_package:build_pip_package)' --output graph > tensorflow.dependency.dot
There are now three main options for building and/or running TensorFlow on Windows:
You can install a GPU-enabled PIP package of TensorFlow 0.12rc0 from PyPI: pip install tensorflow-gpu
You can build the GPU-enabled PIP package yourself using the experimental CMake build. This also gives you the ability to work with TensorFlow in Visual Studio. The documentation for this build can be found here.
There is preliminary support for building TensorFlow using Bazel for Windows. However, we are still ironing out some bugs with this build.
This may not be exactly what you want one way to run TensorFlow under Windows is to install a virtual machine (VMWare player v12 is free to use for non-commercial) and then install Ubuntu in that and finally TensorFlow in Ubuntu. Works well for me.
Since the begin of 2017, Tensorflow is now officially supported on Windows and can be installed via pip:
pip install --upgrade tensorflow
pip install --upgrade tensorflow-gpu
or by fetching packages directly (pick the one that matches your needs, e.g. x64/gpu)
# x86 / CPU
pip install --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.0.0-cp35-cp35m-win_x86_64.whl
# x64 / CPU
pip install --upgrade https://storage.googleapis.com/tensorflow/windows/cpu/tensorflow-1.0.0-cp35-cp35m-win_amd64.whl
# x64 / GPU
pip install --upgrade https://storage.googleapis.com/tensorflow/windows/gpu/tensorflow_gpu-1.0.0-cp35-cp35m-win_amd64.whl
I need to install Numpy on my Mac OS X Lion 10.7. I googled a little bit, it seems like there are a lot of stuff needs to be installed. And some says I need install Xcode before I install Numpy. But the official website of Numpy doesn't give much information about installing Numpy on Mac. I really don't want to install Xcode as it would cost a lot space.
So anyone knows how to install Numpy? What's the prerequisites?
Thanks,
Lion comes with numpy installed. It should already be there:
$ python -c 'import numpy, numpy.version; print numpy, numpy.version.version'
<module 'numpy' from '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/__init__.py'> 1.5.1
For anyone that hits this from google, here is where I figured out how to build numpy on Mac OS.
http://www.scipy.org/Installing_SciPy/Mac_OS_X
On lion, it amounts to setting some environment variables (after Xcode is installed):
export CC=gcc-4.2
export CXX=g++-4.2
export FFLAGS=-ff2c
Then pip install numpy works just fine.
I have used a few software packages that require a fresh install of NumPy, that won't work with the built in NumPy or even with the built-in Python. The easiest way to install NumPy on the mac is to download any of these packages. This will work for most OS versions.
Python 2.7
Python 2.6
Python 2.5
Still the other answers are valid, though this is a more general answer.
This is the SourceForge page of the project.
If you do install Xcode you can get the latest and greatest open source libraries and packages. Consider using the excellent Mac Ports (http://www.macports.org) project. It is a package manager that will download source packages and their dependencies and build all the libraries and tools for you.
Installing Numpy becomes a one line command once Xcode and mac ports is installed:
sudo port install py27-numpy
The easiest way that I found was to visit http://sourceforge.net/projects/numpy/files/NumPy/, pick the folder of the latest version, and then download the .dmg file that matched my installed version of Python.
Based on this post, I found that Chris Fonnesbeck keeps a highly functional installer script for OSX 10.7 at the OSX SuperPack Github site.
Because I hapened to have all the dependencies (such as gfortran) already, I did not use his full shell script. Rather I just did
git clone --depth=1 git://github.com/fonnesbeck/ScipySuperpack.git
python -m easy_install -N -Z ScipySuperpack/*.egg
And it worked beautifully. In particular, it worked better than the .dmg files from the official numpy and scipy distributions, and better than pip install.
You can install numpy/scipy using pip, If pip is not installed on your machine, you can install it with easy_install:
sudo easy_install pip
Then install numpy:
sudo pip install numpy
A very easy route is to install Anaconda from Continuum, or Enthought's Canopy. These are python distributions that include numpy, as well as other useful scientific computing libraries
Generally, unless you have a good reason you might want to avoid trying to have to build numpy/scipy et al. yourself.
For Mac's there are two nice everything-included packages for numerical python. Enthough distribution (paid, http://www.enthought.com/) and Sagemath (free, http://www.sagemath.org/). I'm happily using Sagemath on 10.7 - it has numpy, scipy etc.