Dependency issue with conda, pip and virtualenv - pip

I am using conda(anaconda 2). Before installing anaconda, my Ubuntu16.04 already had pip installed. I recently installed virtualenv using pip install virtualenv. I did the following steps while trying to install coala in my virtual environment.
Create a new virtual environment using virtualenv venv
Activate environment using source activate path/to/activate
Clone the coala repository.
Run pip3 install -e . while inside the repository
After all this, I get the following error from coala:
There is a conflict in the version of a dependency you have installed and the requirements of coala. This may be resolved by creating a separate virtual environment for coala or running `pip install "yapf~=0.14.0"`. Be aware that the latter solution might break other python packages that depend on the currently installed version.
I am already in a virtualenv, so I tried installing yapf as per the command: pip install "yapf~=0.14.0". After that, when I check my pip list, it still showed yapf (0.15.2), To solve this, I did:
pip uninstall yapf
pip install "yapf~=0.14.0"
Now when I check my pip list, I get the desired results. BUT, the error is still there. It still shows the same dependency error. I am unable to understand what is happening? Which pip is installing what and where, what is conda doing behind the scenes, how is virtualenv behaving with conda, and the most important, why this error and how to resolve it?

first create a virtualenv pertaining to python3.x
template
virtualenv -p {location of python3 version } {name of virtualenv folder}
like this
virtualenv -p /usr/local/bin/python3 venv
then activate it
source venv/bin/activate
then run
pip install {library you want to use}
pip install "yapf~=0.14.0"
This should install yapf into the venv folder.
try that let us know.

Answering my own question, what I found out was that conda and virtualenv do not go hand in hand. condas has the ability to create it's own virtual environment, and if using condas, we must create a conda virtual environment(See this answer).
What I did was uninstalled conda and make a virtual environment using virtualenv. What I could have done as well is uninstall virtualenv and create condas environment and work in that.

Related

How to upgrade pyenv (macOS) such that a fresh .venv contains an up-to-date pip?

I'm trying to prevent this warning every time I create a fresh .venv:
> /Users/pi/.pyenv/versions/3.10.0/bin/python -m venv .venv
> . .venv/bin/activate
> pip install ipykernel # or anything
WARNING: You are using pip version 21.2.3; however, version 22.2.2 is available.
You should consider upgrading via the '/Users/pi/code/foo/.venv/bin/python -m pip install --upgrade pip' command.
Somehow pyenv has populated my fresh .venv with an out-of-date pip.
If I execute the suggested command it will upgrade my .venv's pip. But I don't want to be doing that every time I create a .venv.
I figured this might fix it, but it doesn't:
> /Users/pi/.pyenv/versions/3.10.0/bin/python -m pip install --upgrade pip
Requirement already satisfied: pip in /Users/pi/.pyenv/versions/3.10.0/lib/python3.10/site-packages (22.2.1)
Collecting pip
Using cached pip-22.2.2-py3-none-any.whl (2.0 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 22.2.1
Uninstalling pip-22.2.1:
Successfully uninstalled pip-22.2.1
Successfully installed pip-22.2.2
What is actually happening when I execute the above command? I was expecting it to update the pip for the python version created/maintained by pyenv. Which it seems to be doing:
🧢 pi#pPro18-4 ~/.pyenv/versions/3.10.0
> find . -name 'pip*'
./bin/pip3
./bin/pip
./bin/pip3.10
./lib/python3.10/site-packages/pip
./lib/python3.10/site-packages/pip-22.2.2.dist-info
🧢 pi#pPro18-4 ~/.pyenv/versions/3.10.0
> ./bin/pip --version
pip 22.2.2 from /Users/pi/.pyenv/versions/3.10.0/lib/python3.10/site-packages/pip (python 3.10)
So why isn't this pip getting copied into my .venv when I create it?
I thought that was the way .venv creation worked.
How to clean up my pyenv Python installation so that it spawns up-to-date .venvs?
EDIT:
Insight from #python on IRC/Libera:
grym: I don't think you can; i just get in the habit of python -m venv somevenv && somevenv/bin/python -m pip install --upgrade pip setuptools wheel
jinsun: python -m venv --upgrade-deps .venv is a simple solution if you were just annoying by the pip warning (...) it is updating the pip inside the venv, forget about the base python, I don't even have pip in the base python
This is the use case for pyenv-hooks
pyenv-hooks are scripts that are executed by pyenv whenever certain commands are run. You can create hooks for regular commands like: exec, rehash, which, but it can also be a plugin command, like virtualenv. The scripts can be written in any language.
Here is the wiki with official instructions.
You can have a hook by creating a script at the following location:
$PYENV_ROOT/pyenv.d/<hook-name>/<your-script-name>
For example, to create a hook that upgrades pip, create a new script within this path:
$PYENV_ROOT/pyenv.d/virtualenv/after.bash
With contents:
after_virtualenv 'PYENV_VERSION="$VIRTUALENV_NAME" pyenv-exec pip install --upgrade pip'
after_virtualenv is the command that tells pyenv when to execute. First, it sets the pyenv version to the name of the virtualenv we just created. with the variable $VIRTUALENV_NAME. Then it upgrades pip itself.
More details in this article.
I originally posted it as a comment, but was suggested to make it a proper answer.
An easier approach is to use the upgrade-deps flag when you create a virtual environment. Like this:
python3 -m venv --upgrade-deps .venv
It was added on python3.9, and according to the official docs:
--upgrade-deps
Upgrade core dependencies (pip, setuptools) to the latest version in PyPI
So, in other words, it will install pip and upgrade right away.

How can I manually update pip on all pyenv virtualenvs?

I use a different virtual environment for every project I work on (using pyenv and pyenv-virtualenv), and I often come across a message like then when installing a python package using pip:
WARNING: You are using pip version 21.1.3; however, version 21.2.4 is available.
You should consider upgrading via the '/Users/.../.pyenv/versions/3.9.6/envs/sdge/bin/python -m pip install --upgrade pip' command.
However, when I run the pip install --upgrade pip command, it only upgrades pip in that particular virtual environment (which is expected, and that is behavior that I want), and it is annoying to see this across all my different virtual environments.
Is there some kind of shortcut (either with pyenv, pip, or the shell) that I could use to run pip install --upgrade pip on every virtual environment?
I don't use pyenv so I created this code after reading documentation and source code (not tested):
for env in $(pyenv virtualenvs --bare --skip-aliases); do
pyenv activate $env
pip install --upgrade pip
pyenv deactivate
done

How to avoid pip install package again while conda install was done before?

guys:
I use conda install tensorflow-gputo install tensorflow 2.0 , and
numpy=1.20.2 would be one of the package installed, and then I use python3 -m pip install SOMEPACKAGE ,this SOMEPACKAGE needs numpy to be installed as well , but pip seems does not check or realize the package numpy has already installed...
I would like to show everything I know so far :
1.I know the packages installed via conda install would go to anaconda3/envs/YOUR_ENV/lib/site-packages
2.I use python3 -m pip install -t anaconda3/envs/YOUR_ENV/lib/site-packages to force the package would be installed to the place where conda install would be.
However,pip still tries to dwonload *.whl file and install package again,I do not want this package installation process happen again ,while it did mention that I can use --upgrade to replace the existed package...
So I would like to know
How does pip and conda install check if the target package has already existed before they actually to through install process?
I think using python3 you are not using interpreter from your current conda environment so it gets installed elsewhere
python -m pip install (or simply pip install) from your activated environment should work and ignore dependencies installed by conda if they satisfy the requirements

pip install -r requirement.txt on virtualenv doesn't install locally but globally, even after activation

I tried to install all dependencies in my requirements.txt (a bunch of packages list).
What I did:
created virtual env virtualenv my_env
activated the new virtual env, I'm able to see my virtual env before the prompt (my_env) $
ran pip install -r requirements.txt
All packages got installed, but when checking with pip freeze I have nothing. Tried to deactivate the virtual env, and made pip freeze again, here I have all installed.
I'm a bit confused because, I'm very sure my virtual env was activated, and I have the right pip path when doing which pip inside it (/home/virtual_env/my_env/bin/pip). Plus, I tried to install one by one the dependency, and they got installed right inside the virtual env & displayable with pip freeze
I cannot do all of them one by one, and I need to reproduce the installation somewhere. Could someone helps on this?
Still no clean solution for this so far, but what would work is to copy-edit (search & replace the return character in the requirements.pip to && pip install. Meaning, edit it from this format
package1==vX.Y
package2==vU.V
...
into this
package1==vX.Y && pip install package2==vU.V ...
Add pip install at the beginning then make a copy to all for install command like
pip install package1==vX.Y && pip install package2==vU.V ...

Installing Python Package to Conda Env Without Using Conda

I have a fresh Anaconda environment in which I would like to install the latest release of mysql-connector-python (v2.1.3). I'm on a CentOS6 system. The problem is, the newest Conda-hosted package is 2.0.3 and because the connector is not currently hosted on PyPI, it cannot be installed via Pip. (Pip's --allow-external option has been deprecated.)
I know that I can easily install the package via yum install, but I believe that will install it outside of the Conda environment.
Is there any way, using yum or otherwise, to isolate this package solely to this particular Conda environment?
Turns out this is surprisingly simple. Just download the source for the Python package that is not available on PyPI and pip install at the source directory. Just remember to have the Conda environment you want to install to active and Pip will correctly isolate the install to env scope.
I was unaware of Pip's install from local options, so big thanks to the folks on the Anaconda mailing list for their help with this.

Resources