In pip there is an -f option which does the following:
-f, --find-links : If a url or path to an html file, then parse for links to archives. If a local path or file:// url that's a directory, then look for archives in the directory listing.
This is the preferred way of installing PyTorch, by setting the link to their overview website, e.g.:
pip3 install torch===1.3.0 -f https://download.pytorch.org/whl/torch_stable.html
For my virtual environments I use pipenv but I haven't found an option that does the same as -f. In the meantime, I can just look up the direct link to the package that is relevant for my system, but that is cumbersome.
Does pipenv provide a way to do the same thing as pip's -f?
In the new version of pipenv (I tested with version 2020.11.15), you can install packages in this way:
pipenv install https://download.pytorch.org/whl/cpu/torch-1.3.0%2Bcpu-cp36-cp36m-linux_x86_64.whl
The link can be found in the this page: https://download.pytorch.org/whl/torch_stable.html
This will be added to the Pipfile as well.
[packages]
torch = {file = "https://download.pytorch.org/whl/cpu/torch-1.3.0%2Bcpu-cp36-cp36m-linux_x86_64.whl"}
you need to manually check the link with your compute platform, os and python version.
It is possible to use environment variables recognised by pip to tweak its behaviour within pipenv's execution. E.g.:
PIP_FIND_LINKS=https://download.pytorch.org/whl/torch_stable.html pipenv install torch==1.5.1+cu101
See:
Advanced Usage of Pipenv - Configuration With Environment Variables
pip's User Guide - Environment Variables
For now, I haven't found a work-around. What you can do, of course, is enabling the pipenv shell and doing what you must with pip, e.g.
pipenv shell
python -m pip install torch===1.3.0 -f https://download.pytorch.org/whl/torch_stable.html
This will install torch in the pipenv environment but torch will not be added to the Pipfile (nor to the lock file).
Related
Background
I have a multi-stage Dockerfile for a JupyterLab image. There are three stages:
server
kernel: mamba create -y -n /tmp/kernel-packages/myenv ...
runner:
FROM ... as runner
...
COPY --from=kernel /tmp/kernel-packages/ /opt/conda/envs
RUN conda config --append envs_dirs /opt/conda/envs/
...
Problem
In the resulting image, the preferred python -m pip works, but pip gives:
bash: /opt/conda/envs/myenv/bin/pip: /tmp/kernel-packages/myenv/bin/python3.9: bad interpreter: No such file or directory
The reason is that pip has #!/tmp/kernel-packages/myenv/bin/python3.9 as its shebang.
Expected
A behaviour like pip install --root /tmp/server-packages ..., which works perfectly fine in a COPY --from=server /tmp/server-packages /.
Additional information
Additionally, some other binaries, like curl or papermill also have wrong paths hardcoded by conda. I've read about Anaconda | Moving Conda Environments, but it seems like overkill to use conda-pack and conda-unpack.
Workaround
Simply create an env by name: mamba create -y -n myenv and then in the runner stage COPY --from=kernel /opt/conda/envs /opt/conda/envs.
Question
Is there a Conda (Mamba) equivalent to pip install's --root option? I'm not looking for a solution to the stated problem as I have I have already found a way that works for me. My interest is purely in the functionality of the conda binary.
The --prefix argument is the equivalent - just that some Conda packages use hardcoded paths, hence the issue you encounter.
conda-prefix-replacement
To properly move a Conda environment to a new prefix via a COPY operation one would need to run the conda-prefix-replacement tool (a.k.a., cpr) to ensure that all files with hardcoded paths get updated to the new location. Presumably, conda-pack is doing a similar thing, just under-the-hood.
For your purposes, you might consider pre-running cpr on the environment(s) in the kernel image so that they are ready to work in the deployed location. Though that would mean always COPYing to the same location.
See the cpr repository for details on use.
I am running
poetry install
from within a python local virtualenv ".venv" . The project is supposed to create an executable hercl that becomes available on the user's path. Two questions:
What options / configuration of I'm not sure if that's supposed to gets installed into the local .venv/bin or in the pyenv shims.
Since poetry reuses / redirects many functions to pip it may be the case that the feature I'm asking about is actually from pip itself. I have not been able to discover from either poetry or pip documentation about this shell script installation. How is this achieved?
Update
After running running pip install outside of the virtualenv it pulls from pypi and creates a bash script ~/.pyenv/shims/my_app .
In my case the my_app is "hercl" and we see this:
$which hercl
~/.pyenv/shims/hercl
Its contents are :
$cat $(which hercl)
#!/usr/bin/env bash
set -e
[ -n "$PYENV_DEBUG" ] && set -x
program="${0##*/}"
export PYENV_ROOT="~/.pyenv"
exec "~/.pyenv/libexec/pyenv" exec "$program" "$#"
Somehow this script is installed when running pip install: I am wondering how pip knows to do this. Is it from the pyproject.ml from poetry ? Is it from a setup.py or setup.cfg associated with pip ?
Anoterh Update #sinoroc has another tack on this: poetry has a scripts section that I did not notice (noobie on that tool).
[tool.poetry.scripts]
hercl = "hercl.hercl:main"
hercl is a command that I was looking for .
But there was also an actual _bash script that would launch hercl that got installed under the shims as part of the virtualenv. i think that script were in the
In a Poetry-based project such executable scripts are defined in the scripts section of pyproject.toml.
If a virtual environment is active when installing the application then the executable is installed in the virtual environment's bin directory. So it is available only while the virtual environment is "active".
I tried to create virtual environment using pyenv virtualenv 3.8.2 myenv, but it failed, i don't know why, i even changed the python version or the virtual environment name, but it still not working.
Some detail:
I tried this in macos and zsh.
➜ ~ pyenv virtualenv 3.8.2 myenv
Looking in links: /var/folders/_9/l8m14fgs6ts9wx0nl1qbzbkm0000gn/T/tmpe9l25o2_
Requirement already satisfied: setuptools in /Users/fitz/.pyenv/versions/3.8.2/envs/myenv/lib/python3.8/site-packages (41.2.0)
Requirement already satisfied: pip in /Users/fitz/.pyenv/versions/3.8.2/envs/myenv/lib/python3.8/site-packages (19.2.3)
rm: /Users/fitz/.pyenv/shims/shims: is a directory
➜ ~ pyenv versions
system
3.7.3
* 3.8.2 (set by /Users/fitz/.python-version)
3.9.5
I had the same issue. For me it was caused by two empty folders in my shims directory. I deleted the folders and it fixed the issue. So, just running the following would fix your issue.
rm -rf ~/.pyenv/shims/shims
rm -rf ~/.pyenv/shims/versions
Try using 'venv' instead of pyenv or pyvenv or virtualenv. Venv is a library that already comes with your python installation. Virtualenv is an external one.pyenv is similar to venv in that it lets you manage multiple python environments. However with pyenv you can't conveniently rollback library installs to some start state and you will likely need admin privileges at some point to update libraries. So I think it is also best to use venv.
First, make a directory :
mkdir testing
Then, moved to this directory named testing :
cd testing
When you type following command in this directory:
python3 -m venv env
You got error like :
The virtual environment was not created successfully because ensurepip is not
available. On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.
apt install python3.8-venv
Type the following command but before that keep an eye on the version of python you installed on the machine; in my case its python3.8
sudo apt install python3.8-venv
Now, we can create a virtual environment and store its tools in the "bhandari" folder .
python3 -m venv bhandari
Note: you can named this "bhandari" folder; anyname you like( Standard practice is to name it "env" ...)
Now to activate your virtual environment, from the directory of your folder, type the following command this will activate our virtual environment in the “bhandari” folder
source bhandari/bin/activate
If you have successfully activated your virtual environment, you should see the (bhandari) word indicating that we are working in a virtual environment.
After this, we can install anything that will be isolated from the rest of the system....
To install to my own directory I can use
pip install --user package
Alternatively I can use
conda install package
How do I ask conda to install to my home directory since conda does not take a --user flag?
Installing pip packages to $HOME folder
I don't think it's possible. Use virtual environments (conda create).
See -p option here:
-p PATH, --prefix PATH
Full path to environment prefix.
So to install to, say, local under your home directory, do:
conda install -p $HOME/local packagename
Note, however, this is not a "clean" install as it adds a bunch of conda-related files.
To install conda packages on your own directory you can follow these steps:
Create a blank environment
conda create -y -n my-conda-env
Replace the name my-conda-env with any name you want to give the environment.
Activate the environment
source activate my-conda-env
Don't forget to replace my-conda-env with the name you gave the conda environment from the previous step
Install your package
conda install -c bioconda epa-ng
And that's it, the package should be installed on your own directory
Simply:
sudo conda install -c conda-forge package
Or:
sudo chmod -R 777 ./
conda install -c conda-forge package
I don't know of an exact match for the --user flag, but a reasonable analogue is to use a virtual environment.
What I do when I have to install to a shared CentOS server where I don't have admin access:
First I run
conda env list
will list all conda virtual environments and display the path to each. Once you have the environment created and can see it in the conda env list, copy the path to the environment.
If you need to create one, you can do that with conda create or by running anaconda-navigator and using the GUI.
Activate your environment (if not active) with
conda activate [environment_name]
or
activate [environment_name]
depending on your system (most linux systems use the first, Windows and CentOS use the latter).
Now you can use
conda install -p [environment_path] [package_name]
and you are off to the races.
This is really a work around; it's not the best but it does install the package to the selected virtual environment.
The current Anaconda Install Individual Edition, when run in a linux local account, installs in a local directory. So all the subsequent installs should install there, too.
According to the documentation:
--use-local
I have installed some packages with -e
> pip install -e git+https://github.com/eventray/horus.git#2ce62c802ef5237be1c6b1a91dbf115ec284a619#egg=horus-dev
I with pip freeze I see
> pip freeze
...
-e git+https://github.com/eventray/horus.git#2ce62c802ef5237be1c6b1a91dbf115ec284a619#egg=horus-dev
...
when I try to uninstall the packages I get errors:
> pip uninstall horus-dev
Cannot uninstall requirement horus-dev, not installed
> pip uninstall horus
Cannot uninstall requirement horus, not installed
How do I uninstall such a package?
At {virtualenv}/lib/python2.7/site-packages/ (if not using virtualenv then {system_dir}/lib/python2.7/dist-packages/)
remove the egg file (e.g. distribute-0.6.34-py2.7.egg) if there is any
from file easy-install.pth, remove the corresponding line (it should be a path to the source directory or of an egg file).
An easier way to do the same with the new version of setup_tools is to run the following:
python setup.py develop -u
Which basically does the same as what #glarrain describes in his answer.
Here's a demonstration, showing that eg you don't want to substitute a package name into that command:
.../pytest-migration$ python setup.py develop -u
running develop
Removing /home/me/virtualEnvs/automation/lib/python2.7/site-packages/pytest-migration.egg-link (link to .)
Removing pytest-migration 1.0.155 from easy-install.pth file
.../pytest-migration$
Install a dev package use cmd:
pip install --editable .
Uninstall:
rm -r $(find . -name '*.egg-info')
Now you can use:
pip uninstall package_name
or python setup.py develop --uninstall or python setup.py develop -u
Simply uninstall the package you installed in 'editable' mode:
pip uninstall yourpackage
it works for recent pip-versions (at least >=19.1.1).
It turns out that my installation was somehow corrupt.
I could find the entry in:
/usr/local/lib/python2.7/site-packages/easy-install.pth
To solve the problem I removed the line in the .pth file by hand!
import sys; sys.__plen = len(sys.path)
...
/absolute-path-to/horus # <- I removed this line
...
This is a bug on debian/ubuntu linux using OS-installed pip (v8.1.1 for me), which is what you'll invoke with sudo pip even if you've upgraded pip (e.g. get-pip.py). See https://github.com/pypa/pip/issues/4438
For a discussion on how to clean up see https://askubuntu.com/questions/173323/how-do-i-detect-and-remove-python-packages-installed-via-pip, though the solutions there are of the "remove everything" variety.
...pip packages [go] to /usr/local/lib/python2.7/dist-packages, and apt packages to /usr/lib/python2.7/dist-packages
...a few packages were installed in ~/.local/lib too.
For my system all I needed to remove was /usr/local/lib/python2.7/dist-packages/{package_name}.egg-link
I think I have something to add to all the answers here:
Using pip list you'll see all your installed packages, and there is a little trickery: a single pip install can create several entries in this list. In particular when you do an editable install, you'll have your <package_name> listed besides the location of the source on your disc.
This <package_name> is only used for pip and is never called in python as far as I understand, it is configured in your pyproject.toml, setup.cfg or setup.py.
Thus, to properly uninstall your package using pip, you should use this name and not the named of individual modules included in your package.
Hope it helps!