requirements.txt pytorch with ">=" greater than? - pip

I have a requirements.txt from a Github repo that contains following lines:
torch>=1.7.0,!=1.12.0
torchvision>=0.8.1,!=0.13.0
// and more
As I search SO and google they say I need to install pytorch with cuda specified, e.g. +cu110; in order to enable GPU and use the installed cuda.
So, for example this command does work on CLI: pip install torch==1.7.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html
But the problem is with requirements.txt.
I looked Install PyTorch from requirements.txt - Stack Overflow and tried some solutions but they didn't work as the following.
// simply added `+cu110`
// didn't work
torch>=1.7.0+cu110,!=1.12.0
torchvision>=0.8.1+cu110,!=0.13.0
// w/ --extra-index-url
// didn't work
--extra-index-url https://download.pytorch.org/whl/cu110
torch>=1.7.0+cu110,!=1.12.0
--extra-index-url https://download.pytorch.org/whl/cu110
torchvision>=0.8.1+cu110,!=0.13.0
// w/ -f
// didn't work
-f https://download.pytorch.org/whl/torch_stable.html
torch>=1.7.0+cu110,!=1.12.0
-f https://download.pytorch.org/whl/torch_stable.html
torchvision>=0.8.1+cu110,!=0.13.0
So, is it possible to work with the combination of >= and +cu110 in requirements.txt?
Edit
It still doesn't work with no +cu110.
The following doesn't install cuda, but instead installs a normal torch.
--extra-index-url https://download.pytorch.org/whl/cu110
torch>=1.7.0+cu110,!=1.12.0
--extra-index-url https://download.pytorch.org/whl/cu110
torchvision>=0.8.1+cu110,!=0.13.0
So the stdout result is like:
Collecting torch!=1.12.0,>=1.7.0
Using cached torch-1.13.0-cp37-cp37m-win_amd64.whl (167.3 MB)
And pip freeze shows:
torch==1.13.0
torchvision==0.14.0
The following also doesn't install cuda.
-f https://download.pytorch.org/whl/cu110
torch>=1.7.0,!=1.12.0
-f https://download.pytorch.org/whl/cu110
torchvision>=0.8.1,!=0.13.0
The stdout and pip freeze result is the same to the above.
The following also doesn't work, it installs cuda, but it installs a random version of cuda. (it's understandable because I didn't specify a version I prefer)
-f https://download.pytorch.org/whl/torch_stable.html
torch>=1.7.0,!=1.12.0
-f https://download.pytorch.org/whl/torch_stable.html
torchvision>=0.8.1,!=0.13.0
The stdout:
Collecting torch!=1.12.0,>=1.7.0
Using cached https://download.pytorch.org/whl/cu117/torch-1.13.0%2Bcu117-cp37-cp37m-win_amd64.whl (2258.1 MB)
Collecting torchvision!=0.13.0,>=0.8.1
Using cached https://download.pytorch.org/whl/cu117/torchvision-0.14.0%2Bcu117-cp37-cp37m-win_amd64.whl (4.8 MB)
pip freeze result:
torch==1.13.0+cu117
torchvision==0.14.0+cu117

Related

Cannot install bcftools-gtc2vcf-plugin using conda

I have installed bioconda following the instructions at https://bioconda.github.io/user/install.html#set-up-channels. Then I tried
conda install bwa
conda install bcftools
conda install plink2
They all installed fine. However, when I tried
conda install bcftools-gtc2vcf-plugin
or
conda install -c bioconda bcftools-gtc2vcf-plugin
as instructed at https://bioconda.github.io/recipes/bcftools-gtc2vcf-plugin/README.html, I got errors as follows:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- bcftools-gtc2vcf-plugin
Current channels:
- https://conda.anaconda.org/bioconda/osx-64
- https://conda.anaconda.org/bioconda/noarch
- https://conda.anaconda.org/conda-forge/osx-64
- https://conda.anaconda.org/conda-forge/noarch
- https://repo.anaconda.com/pkgs/main/osx-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/osx-64
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Any help would be highly appreciated.
Thanks in advance!
I would advise (as of 2020-01-06) not to use the bcftools-gtc2vcf-plugin as it is an old version missing many features compared to the current version. I would advise either to compile from source (https://github.com/freeseek/gtc2vcf) or alternatively to download pre-compiled binaries (https://personal.broadinstitute.org/giulio/gtc2vcf) that should work on systems with ≥GLIBC_2.3 installed (and making sure you are running the latest version of BCFtools)
If you get the error:
No functional bcftools plugins were found in
BCFTOOLS_PLUGINS="/Users/moxu/xbin/seq/bcftools/plugins".
- Is the plugin path correct?
- Run "bcftools plugin -lv" for more detailed error output.
Could not load "gtc2vcf".
(a bcftools plugin bug that the maintainers will fix soon), can you try to run one of the following commands instead:
$ bcftools plugin gtc2vcf -vv
$ bcftools +gtc2vcf -vv
$ bcftools plugin /Users/moxu/xbin/seq/bcftools/plugins/gtc2vcf.so -vv
$ bcftools +/Users/moxu/xbin/seq/bcftools/plugins/gtc2vcf.so -vv
You should get a reason for why the plugin is not loading. A typical error message could look like this:
/Users/moxu/xbin/seq/bcftools/plugins/gtc2vcf.so:
dlopen .. /lib64/libc.so.6: version `GLIBC_2.3' not found (required by /Users/moxu/xbin/seq/bcftools/plugins/gtc2vcf.so)

Trying to Serve a ML (pytorch) model on a Heroku, slug size too large [duplicate]

I've been getting the slug size too large warning (Compiled slug size: 789.8M is too large (max is 500M)) from Heroku and I can't figure out why, as my model size (cnn.pth below) is fairly small and my file directory is only 1.1mb in total: screenshot of directory.
It seems like the size increase is caused by running pipenv install torch, as the slug size was 89.1mb before installing torch and 798.8mb after.
My Pipfile currently has these packages installed:
[packages]
flask = "*"
flask-sqlalchemy = "*"
psycopg2 = "*"
psycopg2-binary = "*"
requests = "*"
numpy = "*"
gunicorn = "*"
pillow = "*"
torch = "*"
Is there any workaround for this?
Edit: I'm running Mac OSX 10.10.5, using Flask and pipenv.
The pytorch package that you're installing comes with both cpu and gpu support, thus has a large size. It seems you're using the free version of heroku, and only require the cpu support. The solution is to install the pytorch package for cpu only i.e.
In requirements.txt, write the wheel file path corresponding to the version of pytorch (cpu) that you're interested in. You can find the list of wheel files, which can be installed with pip. For example, for PyTorch 1.3.1, torchvision 0.4.2, Python 3.7, Linux, you can write the following for pytorch and torchvision respectively:
https://download.pytorch.org/whl/cpu/torch-1.3.1%2Bcpu-cp37-cp37m-linux_x86_64.whl
https://download.pytorch.org/whl/cpu/torchvision-0.4.2%2Bcpu-cp37-cp37m-linux_x86_64.whl
The above will download torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl (107MB) torchvision-0.4.2+cpu-cp37-cp37m-linux_x86_64.whl (13MB) respectively.
Go to PyTorch Get Started page. Choose Stable version, Linux, pip, python, cpu and you can see:
pip install torch==1.8.1+cpu torchvision==0.9.1+cpu torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
Copy versions to your requirements.txt.
torch==1.8.1+cpu
torchvision==0.9.1+cpu
And add to the beginning of the requirements.txt this:
-f https://download.pytorch.org/whl/torch_stable.html
After that, all packages will be installed on Heroku.
For reference:
torch==1.8.1+cpu (169.1 MB);
torchvision==0.9.1+cpu (13.3 MB)
By today, March 28, 2022, these are the working versions:
torch==1.11.0+cpu
torchvision==0.12.0+cpu
(Aug, 2, 2022) the only solution I found was leaving the requirements.txt like this:
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==1.11.0+cpu
--find-links https://download.pytorch.org/whl/torch_stable.html
torchvision==0.12.0+cpu

pyethapp installation issue on OSX 10.13.2

Installing pyethapp in virtual environment on OSX 10.13.2 with following properties:
Python 2.7.14 pip (9.0.1) setuptools (38.2.4) wheel (0.30.0)
and get following error message:
Collecting pyethapp
Using cached pyethapp-1.5.0-py2.py3-none-any.whl
Collecting ethereum>=1.5.1 (from pyethapp)
Using cached ethereum-2.2.0.tar.gz
Complete output from command python setup.py egg_info:
error in ethereum setup command: 'tests_require' must be a string or list of strings containing valid project/version requirement specifiers; Unordered types are not allowed
Any idea how to resolve this?
I have the same issue on Arch Linux.
It looks like setup.py tests_require uses a set which is not support in setuptools > 38.0.
It should be.
pip install setuptools==37
This trick works for me.
Greetings.

Why can't pip find pysvn?

I'm working on a project which was written in Python 2, and I'm upgrading it to Python 3. So far, I've just been finding minor syntax errors which are easily fixable. What I've done is created a new project in Python 3, ensured that it worked, and copies chunks of code from the old project into the new one.
Right now, I'm having trouble with pysvn. Initially, I was getting this error:
ImportError: No module named 'pysvn'
At this point, I tried using pip install pysvn, which didn't work. I got the following:
pip install pysvn
Collecting pysvn
Could not find a version that satisfies the requirement pysvn (from versions:)
No matching distribution found for pysvn
So then after a bit of research, I went to the pysvn download site and tried:
>pip install --index-url http://pysvn.tigris.org/project_downloads.html pysvn, which gave me this error:
Collecting pysvn
The repository located at pysvn.tigris.org is not a trusted or secure host and is being ignored. If this repository is available via HTTPS it is recommended to use HTTPS instead, otherwise you may silence this warning and allow it anyways with '--trusted-host pysvn.tigris.org'.
and also the same error as when I tried >pip install pysvn.
My next step was to manually download the .exe file for the version I needed, and I was able to successfully install pysvn. I have checked the site-packages directory, and pysvn is indeed there, but pip still can't tell me anything about it:
>pip show pysvn
>
When I do this for another installed module, selenium for example, I get the following:
pip show selenium
Metadata-Version: 1.1
Name: selenium
Version: 2.49.2
Summary: Python bindings for Selenium
Home-page: https://github.com/SeleniumHQ/selenium/
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: ...\lib\site-packages
Requires:
I was able to verify that the installation of pysvn was successful because my project now runs instead of giving me that ImportError.
So why can pip not give me information for another module in the same directory that was successfully installed?
As it turns out, because I didn't use pip install for pysvn, pip didn't know that pysvn existed. Because it wasn't available from PyPI (the Python Package Index), there was no way that pip could see it (because that's where pip goes first to find packages that it's attempting to install).
From the pip user guide:
pip supports installing from PyPI, version control, local projects, and directly from distribution files.
Since I had eventually downloaded pysvn from its own download site (which was not any of the above 4 options) and ran the .exe manually, pip simply doesn't know about it even though it's in the same directory as other packages installed by pip.
I suppose I could've also retrieved the distribution files and used pip with those, but my workaround did the trick.
My way on linux:
Get sources from here
tar -zxf pysvn-1.9.10.tar.gz
apt-get install subversion libsvn1 libsvn-dev make g++
cd pysvn-1.9.10/Source
python setup.py configure --pycxx-dir=/pysvn-1.9.10/Import/pycxx-7.1.3/
make
Here i've got errors below:
Compile: /pysvn-1.9.10/Import/pycxx-7.1.3/Src/cxxsupport.cxx into cxxsupport.o
/pysvn-1.9.10/Import/pycxx-7.1.3/Src/cxxsupport.cxx:42:10: fatal error: Src/Python3/cxxsupport.cxx: No such file or directory
#include "Src/Python3/cxxsupport.cxx"
Compile: /pysvn-1.9.10/Import/pycxx-7.1.3/Src/cxxextensions.c into cxxextensions.o
/pysvn-1.9.10/Import/pycxx-7.1.3/Src/cxxextensions.c:42:10: fatal error: Src/Python3/cxxextensions.c: No such file or directory
#include "Src/Python3/cxxextensions.c"
It is needed to edit that files:
vi /pysvn-1.9.10/Import/pycxx-7.1.3/Src/cxxsupport.cxx
change #include "Src/Python3/cxxsupport.cxx" to
#include "Python3/cxxsupport.cxx"
and same on second file. Than make again:
make clean && make
...
Compile: /code/pysvn-1.9.10/Import/pycxx-7.1.3/Src/cxxextensions.c into cxxextensions.o
Compile: /code/pysvn-1.9.10/Import/pycxx-7.1.3/Src/IndirectPythonInterface.cxx into IndirectPythonInterface.o
Compile: /code/pysvn-1.9.10/Import/pycxx-7.1.3/Src/cxx_exceptions.cxx into cxx_exceptions.o
Link pysvn/_pysvn_3_7.so
Then just copy it to the site-packages (change to yours directory):
mkdir /usr/local/lib/python3.7/site-packages/pysvn
cp /code/pysvn-1.9.10/Sources/pysvn/__init__.py /usr/local/lib/python3.7/site-packages/
cp /code/pysvn-1.9.10/Sources/pysvn/_pysvn*.so /usr/local/lib/python3.7/site-packages/

Installing PyAudio for Python 3 on OS X

I would like to construct a "Hello, World!" of audio for OS X/Python 3 that populates a buffer with a sine wave and plays it through the speakers.
My basic options are listed in Audio in Python.
Playing Music with Python demonstrates several of these, but it omits details of installation.
I'm trying PyAudio first.
PyAudio has a section on OS X, saying "For Python 3 support, first install MacPython 3.3"
This confuses me greatly. OS X ships with Python. Also, I have used Homebrew to install Python 3. I've never come across the term MacPython, so I'm not sure if what I have is MacPython or not. And if it isn't, then I want to install PyAudio for the Python I currently have. I don't want to have to download some special Python to use it. That would be completely defeating the purpose.
I also tried "pip3 install pyaudio", with the following (negative) results:
⤐ pip3 install pyaudio
Collecting pyaudio
Could not find a version that satisfies the requirement pyaudio (from versions: )
Some externally hosted files were ignored as access to them may be unreliable (use --allow-external pyaudio to allow).
No matching distribution found for pyaudio
✘
pi#piBookAir.local ~ /Users/pi:
⤐ pip install --allow-external pyaudio
You must give at least one requirement to install (see "pip help install")
✔
pi#piBookAir.local ~ /Users/pi:
⤐ pip install pyaudio --allow-external pyaudio
Collecting pyaudio
Could not find a version that satisfies the requirement pyaudio (from versions: )
Some insecure and unverifiable files were ignored (use --allow-unverified pyaudio to allow).
No matching distribution found for pyaudio
✘
It's frustrating that it is so awkward to even get onto the first rung of the ladder.
How can I correctly install PyAudio for Python 3 on my (up-to-date) OS X? Should I use Homebrew? Should I use pip3?
PS: PyAudio with Homebrew -- I hope I don't have to follow this solution
PPS: Music software written in Python lists a daunting number of possible solution paths -- I'm not at all certain PyAudio is the best path. Is there a better one?
Use Homebrew to install the prerequisite portaudio library, then install PyAudio using pip:
brew install portaudio
pip install pyaudio
Notes:
If not already installed, download Homebrew.
pip will download the PyAudio source and build it for your version of Python.
Homebrew and building PyAudio also require installing the Command Line Tools for Xcode (more information).
(This works on Mac OS X)
At first PyAudio has a portaudio dependency:
brew install portaudio
And then run:
pip3 install pyaudio --global-option="build_ext" --global-option="-I/usr/local/include" --global-option="-L/usr/local/lib"
Or sometimes magic happens, if you use:
python3 -m pip install pyaudio --global-option="build_ext" --global-option="-I/usr/local/include" --global-option="-L/usr/local/lib"
Unfortunately, 王淳龙's solution did not work for me (macOS v10.15.6 (Catalina), Python 3.8.5). The following errors persisted:
gcc-5 -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/tom/anaconda3/include -arch x86_64 -I/Users/tom/anaconda3/include -arch x86_64 -DMACOSX=1 -I/Users/tom/anaconda3/include/python3.6m -c src/_portaudiomodule.c -o build/temp.macosx-10.9-x86_64-3.6/src/_portaudiomodule.o
In file included from src/_portaudiomodule.c:33:0:
/usr/local/include/pa_mac_core.h:48:33: fatal error: AudioUnit/AudioUnit.h: No such file or directory
compilation terminated.
error: command 'gcc-5' failed with exit status 1
Therefore, I had to link the missing libraries manually:
cd /usr/local/include/
ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/AudioUnit.framework/Versions/A/Headers AudioUnit
ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/AudioToolbox.framework/Versions/A/Headers AudioToolbox
ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/CoreAudioTypes.framework/Versions/A/Headers CoreAudioTypes
ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/CoreFoundation.framework/Versions/A/Headers CoreFoundation
ln -s /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/CoreAudio.framework/Versions/A/Headers CoreAudio
This solved the dependencies, but not the following compiler problem:
In file included from /usr/local/include/AudioToolbox/AUComponent.h:65:0,
from /usr/local/include/AudioUnit/AUComponent.h:1,
from /usr/local/include/AudioUnit/AudioUnit.h:11,
from /usr/local/include/pa_mac_core.h:48,
from src/_portaudiomodule.c:33:
/usr/local/include/AudioToolbox/AudioComponent.h:509:39: error: expected ')' before '^' token
void (^inCompletionHandler)(AudioComponentInstance __nullable, OSStatus))
^
Forcing the use of system-wide gcc (Apple Clang version 11.0.3) instead of gcc-5 solved the problem. I did that by uninstalling gcc#5 and linking gcc-5 to gcc. However, there must be a more elegant way.
Today I also encountered this problem and searched a lot. Finally I found a solution:
brew install portaudio
pip install pyaudio
Python vs MacPython:
AFAIK there is no such thing as MacPython — or at least not from the developers of Python. I can only assume whoever created the link on the PyAudio page tried to get creative with the name and were actually meaning the "Mac OS X 64-bit/32-bit installer containing Python v3.3".
The link to the mysteriously named MacPython leads to the main Python download page that features the most recent stable versions (which as of today is v3.4.3):
https://www.python.org/downloads/release/python-343/
PyAudio:
PyAudio can be compiled from source, outlined here, which might yield better results.

Resources