I have a Virtual env for my django project, but when I hit pip freeze, I get what must be a global site package list, includes too many packages, like ubuntu packages and so much irrelevant stuff. This happens whether virtualenv is active or not. My site packages list looks a bit slim too, so I wonder whether venv has been working at all.
(env)~/code/django/ssc/dev/env/lib/python2.7/site-packages> ls
django
Django-1.4-py2.7.egg-info
easy-install.pth
pip-1.0.2-py2.7.egg
setuptools-0.6c11-py2.7.egg
setuptools.pth
What's my problem?
If your virtual environment has access to the system's site-packages dir (ie. you used virtualenv --system-site-packages) then it's normal for the list to be a rather long one.
Compare the following:
$ virtualenv --system-site-packages v1 && source v1/bin/activate
$ (v1) pip freeze | wc -l # 100
$ virtualenv v2 && source v2/bin/activate
$ (v2) pip freeze | wc -l # 2
Can you try recreating the virtualenv?
Alternatively, adding a no-global-site-packages.txt file should tell pip to ignore the global site-packages:
$ touch $VIRTUAL_ENV/lib/python${version}/no-global-site-packages.txt
I don't understand why the most concise option was just left in the comments. Since I have just almost missed it I will put it here as a separate answer with some tweaks.
You can add --local flag with your pip freeze if you are running a virtual env with system-site-packages enables.
So, if you had:
py -m venv --system-site-packages env
To make sure you are not getting all system deps into your requirements.txt, just run:
python -m pip freeze --local > requirements.txt
Another, a bit more elaborate option, but still viable because dependencies are not supposed to change all that often, would be to go into pyvenv.cfg file located in your virtual env library and manually change:
include-system-site-packages = true/false
Related
Background
I have a multi-stage Dockerfile for a JupyterLab image. There are three stages:
server
kernel: mamba create -y -n /tmp/kernel-packages/myenv ...
runner:
FROM ... as runner
...
COPY --from=kernel /tmp/kernel-packages/ /opt/conda/envs
RUN conda config --append envs_dirs /opt/conda/envs/
...
Problem
In the resulting image, the preferred python -m pip works, but pip gives:
bash: /opt/conda/envs/myenv/bin/pip: /tmp/kernel-packages/myenv/bin/python3.9: bad interpreter: No such file or directory
The reason is that pip has #!/tmp/kernel-packages/myenv/bin/python3.9 as its shebang.
Expected
A behaviour like pip install --root /tmp/server-packages ..., which works perfectly fine in a COPY --from=server /tmp/server-packages /.
Additional information
Additionally, some other binaries, like curl or papermill also have wrong paths hardcoded by conda. I've read about Anaconda | Moving Conda Environments, but it seems like overkill to use conda-pack and conda-unpack.
Workaround
Simply create an env by name: mamba create -y -n myenv and then in the runner stage COPY --from=kernel /opt/conda/envs /opt/conda/envs.
Question
Is there a Conda (Mamba) equivalent to pip install's --root option? I'm not looking for a solution to the stated problem as I have I have already found a way that works for me. My interest is purely in the functionality of the conda binary.
The --prefix argument is the equivalent - just that some Conda packages use hardcoded paths, hence the issue you encounter.
conda-prefix-replacement
To properly move a Conda environment to a new prefix via a COPY operation one would need to run the conda-prefix-replacement tool (a.k.a., cpr) to ensure that all files with hardcoded paths get updated to the new location. Presumably, conda-pack is doing a similar thing, just under-the-hood.
For your purposes, you might consider pre-running cpr on the environment(s) in the kernel image so that they are ready to work in the deployed location. Though that would mean always COPYing to the same location.
See the cpr repository for details on use.
Most of you may know OpenAI playground, so I built an function generator app and followed all the instructions but I can't launch it via venv-python. I can actually install the requirements using pip while venv is not active and can launch the site with flask, but when I try to install requirements after activating venv, I get this error;
User#lalec ~
$ cd openai-quickstart-python
User#lalec ~/openai-quickstart-python (master)
$ . venv/Scripts/activate
(venv)
User#lalec ~/openai-quickstart-python (master)
$ pip install -r requirements.txt
Fatal error in launcher: Unable to create process using '"C:\Users\celal\openai-quickstart-python\venv\Scripts\python.exe" "C:\Users\User\openai-quickstart-python\venv\Scripts\pip.exe" install -r requirements.txt': The system cannot find the file specified.
How do I fix this? I added every possible script locations into PATH I thought it would help but no result.
I just realized two directories in the error doesn't match, and "C:\Users\celal\openai-quickstart-python\venv\Scripts\python.exe" actually does not even exist. Maybe that's the problem... How can I change this ??
Sorry if I'm asking dumb questions I'm new to all this... Also, I don't get why I need venv activated while I can just launch it by accessing the directory and type flask run in cmd, would appreciate any answers.
ss of the project's directory
ss of venv/Scripts/
ss of bash screen with errors
The error message suggests that the requirements.txt file can't be read. Check that you definitely can read it and it's in the expected location.
The perferred command for running pip is:
python3 -m pip install -r requirements.txt
This makes sure that you know which python3 binary you're running pip with - some systems have more than one python3 binary in various locations.
You could achieve similar without activating the virtual environment:
./venv/bin/python3 -m pip install -r requirements.txt
I am trying to get to grips with Python on my Mac running OS X 10.15.7 Catalina.
My goal is to be able to run a program requiring Python 3.9 and associated libraries, in its own virtual environment, using venv.
My quest has led me over many rocky paths and blind alleys - including via Anaconda and Brew.
I am running zsh - specifically oh-my-zsh ys version. If that's important. Most online forums and tutorials assume bash. I would prefer to do all in zsh to avoid having to edit both zsh and bash profiles.
Current Shell Outputs
$ python -V
Python 2.7.16
$ which python
/usr/bin/python
$ python3 -V
Python 3.8.2
$ which python3.8
python3.8 not found
$ which python3.9
python3.9 not found
$ python3 -m -V
/Library/Developer/CommandLineTools/usr/bin/python3: No module named -V
Pausing there ..
Py 2.7 is the OS X system Python and I don't want to do anything with it.
It is odd that python3 -m returns a reference to the CLI tools. See below. The only location for py3.8 is in the CL tools.
What pythons do I have (output below is edited so as to show only main directories and Files)?
$ locate python
/Applications/LibreOffice.app/Contents/Frameworks/LibreOfficePython.framework/Versions/3.7/bin/python3
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
/Library/Developer/CommandLineTools/usr/bin/python3.8
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3-config
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9-config
/Library/Frameworks/Python.framework/Versions/3.9/share/man/man1/python3.1
/Library/Frameworks/Python.framework/Versions/3.9/share/man/man1/python3.9.1
/Library/Frameworks/SAVI.framework/Versions/A/Frameworks/Python.framework/Versions/3.6/lib/python3.6
/usr/bin/python
/usr/bin/python-config
/usr/bin/python2
/usr/bin/python2.7
/usr/bin/python2.7-config
/usr/bin/python3
/usr/bin/pythonw
/usr/bin/pythonw2.7
/usr/lib/libpython.dylib
/usr/lib/libpython2.7.dylib
/usr/lib/python2.7
/usr/local/bin/python3
/usr/local/bin/python3-config
/usr/local/bin/python3.9
/usr/local/bin/python3.9-config
/usr/share/file/magic/python
/usr/share/man/man1/python.1
/usr/share/man/man1/python2.7.1
/usr/share/man/man1/pythonw.1
/usr/share/man/man1/pythonw2.7.1
/usr/share/vim/vim81/autoload/python3complete.vim
/usr/share/vim/vim81/autoload/pythoncomplete.vim
/usr/share/vim/vim81/ftplugin/python.vim
/usr/share/vim/vim81/indent/python.vim
/usr/share/vim/vim81/syntax/python.vim
/usr/share/zsh/5.7.1/functions/_bpython
/usr/share/zsh/5.7.1/functions/_python
/usr/share/zsh/5.7.1/functions/_python_modules
WHAT ABOUT PIP?
$ pip3
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
$ pip3 install --upgrade pip
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in ./Library/Python/3.8/lib/python/site-packages (20.3.3)
Note two points
Although the wrapper warning comes up, PIP 20.3.3 is latest version which I have upgraded. Probably the reason for the warning is in the next lines, which suggest that PIP3 is being invoked from the CL Tools.
2; The ‘normal site-packages is not writeable’ error is supposed to be cured by specifying the Path. I have tried already to specify the correct path - see below - but this has not cured the problem and is another major reason for asking the question at the end of this post.
$ python -m pip
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named pip
$ python3 -m pip
Usage:
/Library/Developer/CommandLineTools/usr/bin/python3 -m pip <command> [options]
$ which pip3
/usr/bin/pip3
$ locate pip3
/Applications/LibreOffice.app/Contents/Frameworks/LibreOfficePython.framework/Versions/3.7/bin/pip3
/Applications/LibreOffice.app/Contents/Frameworks/LibreOfficePython.framework/Versions/3.7/bin/pip3.7
/Library/Developer/CommandLineTools/usr/bin/pip3
/Library/Developer/CommandLineTools/usr/bin/pip3.8
/Library/Frameworks/Python.framework/Versions/3.9/bin/pip3
/Library/Frameworks/Python.framework/Versions/3.9/bin/pip3.9
/usr/bin/pip3
/usr/local/bin/pip3
/usr/local/bin/pip3.9
SYSTEM FILES
~/.zprofile contents - edited a few times to remove references to virfualenv and penv, both originally installed with brew and now removed. I have run brew uninstall and brew remove to try to clean everything up but may not have succeeded.
Start
# THIS FILE COMMENTED OUT RH 21DEC20
## Setting PATH for Python 3.9
## The original version is saved in .zprofile.pysave. **Note in this post only - cannot find this file**
PATH="/Library/Frameworks/Python.framework/Versions/3.9/bin:${PATH}"
export PATH **Note in this post only - this does not seem to have worked - see above Python3 -V**
# export WORKON_HOME=~/.virtualenvs
# mkdir -p $WORKON_HOME
# . ~/.pyenv/versions/3.9.0/bin/virtualenvwrapper.sh
# eval "$(pyenv init -)"
# # AND THESE LINES ADDED FOLLOWING STACKOVERFLOW Q 33216679 **Note in this post only - commented out as shell threw errors .’cannot find virtualenvwrapper.sh**
# export WORKON_HOME=$HOME/.Envs
# # export VIRTUALENVWRAPPER_PYTHON=$/usr/bin/python3
# export VIRTUALENVWRAPPER_PYTHON=$/usr/local/bin/python3
#source $HOME/.local/bin/virtualenvwrapper.sh
END
Finally a word about Brew. Many tutorials advised the use of Brew to install Python, pyenv, and libraries etc. I did that and successfully created a test environment, but could not the utilise it. I have then removed as much as possible in order to get back to a ‘clean’ state to work with.
Current brew status is
$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!
Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.
Unexpected dylibs:
/usr/local/lib/libdvdcss.2.dylib
/usr/local/lib/libfuse.0.dylib
/usr/local/lib/libulockmgr.0.dylib
Warning: Unbrewed header files were found in /usr/local/include.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.
Unexpected header files:
/usr/local/include/ulockmgr.h
Warning: Unbrewed .la files were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.
Unexpected .la files:
/usr/local/lib/libulockmgr.la
Warning: Unbrewed static libraries were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.
Unexpected static libraries:
/usr/local/lib/libfuse.a
/usr/local/lib/libulockmgr.a
SUMMARY
My first goal is to set Py3.9.x as default for new projects. I think also this would require/imply that pip and other commands run from v3.9 and not any other version or location. Must admit to being hazy about that.
Then I want to be able to create a virtual environment into which I can put the program I want to run and its associated libraries and packages. Doing this implies using terminal, but I need to know which directory I need to be in to run which commands.
At this point I have also been running into confusion with tutorials on how to create a virtual environment. From what I have read I think I should use venv over pyenv. What is not clear is whether I run the venv command (s) from root, or whether I have to create and cd to a different location, and if so what.
Any help in
1 Changing the default Python to 3.9
2 Creating a venv to use v3.9.
3 What commands to run where.
Will be greatly appreciated. Please keep it simple and don't assume any prior knowledge!
This is what I would do :
# For 1 Changing the default python to 3.9
cd /usr/local/bin
sudo ln -fs /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 python3
sudo ln -fs python3 python
# And make sure /usr/local/bin is before (/usr/bin and /Library/Developer/CommandLineTools/usr/bin) in your $PATH
# For 2 Creating a venv to use v3.9
/usr/local/bin/python3.9 -m venv /tmp/python3.9-venv
In pip there is an -f option which does the following:
-f, --find-links : If a url or path to an html file, then parse for links to archives. If a local path or file:// url that's a directory, then look for archives in the directory listing.
This is the preferred way of installing PyTorch, by setting the link to their overview website, e.g.:
pip3 install torch===1.3.0 -f https://download.pytorch.org/whl/torch_stable.html
For my virtual environments I use pipenv but I haven't found an option that does the same as -f. In the meantime, I can just look up the direct link to the package that is relevant for my system, but that is cumbersome.
Does pipenv provide a way to do the same thing as pip's -f?
In the new version of pipenv (I tested with version 2020.11.15), you can install packages in this way:
pipenv install https://download.pytorch.org/whl/cpu/torch-1.3.0%2Bcpu-cp36-cp36m-linux_x86_64.whl
The link can be found in the this page: https://download.pytorch.org/whl/torch_stable.html
This will be added to the Pipfile as well.
[packages]
torch = {file = "https://download.pytorch.org/whl/cpu/torch-1.3.0%2Bcpu-cp36-cp36m-linux_x86_64.whl"}
you need to manually check the link with your compute platform, os and python version.
It is possible to use environment variables recognised by pip to tweak its behaviour within pipenv's execution. E.g.:
PIP_FIND_LINKS=https://download.pytorch.org/whl/torch_stable.html pipenv install torch==1.5.1+cu101
See:
Advanced Usage of Pipenv - Configuration With Environment Variables
pip's User Guide - Environment Variables
For now, I haven't found a work-around. What you can do, of course, is enabling the pipenv shell and doing what you must with pip, e.g.
pipenv shell
python -m pip install torch===1.3.0 -f https://download.pytorch.org/whl/torch_stable.html
This will install torch in the pipenv environment but torch will not be added to the Pipfile (nor to the lock file).
I'd like to know how to update anaconda in pyenv, inheriting third-party modules I installed to the former version.
I'm now using anaconda3-2.5.0 and would like to use anaconda3-4.0.0. I manage it with pyenv. If I just install the new version via pyenv, is the third party modules I installed inherited to the new version? How can I do that? Should I be careful to make a list of modules I installed, whenever I install a new module, in order to reinstall them at once to the environment(version).
Sorry for my poor English.
Updating your existing python env isn't something pyenv can help you with, but it can help you manage a new python env using the latest Anaconda, and then you can reformulate that too match your needs.
There are two ways you could approach this problem;
update your anaconda3-2.5.0 in place
create a new anaconda3-4.0.0 and re-install what you need there.
Personally I prefer the second option as you can ensure that you environment is reproducible (for instance if you need to work on a new machine or with another developer), but I'll demonstrate both methods.
1. inplace update
Just use conda to update all your packages (including anaconda):
$ conda update -n <the name of your env> anaconda
$ conda update -n <the name of your env> python package_a package_b ...
etc.
pyenv will still believe that anaconda3-2.5.0 is installed, but you'll know better.
2. new install
First get a record of what you've already installed in your python environment;
$ conda list --export > conda-requirements.txt
$ pip freeze > requirements.txt
Then install the desired (new) python interpreter in pyenv
$ pyenv install anaconda3-4.0.0
and finally re-create your env setup
$ conda create -n <new env name> --file conda-requirements.txt
$ pip install -r requirements.txt