Handling name collisions when using pip with several registries - pip

I have a private registry that stores SomePackage that depends on other packages listed on pypi. I would like to install SomePackage with all its dependencies using pip but I came across a problem.
At first I used the --extra-index-url pip option to install SomePackage:
pip install --extra-index-url http://my.package.repo/simple SomePackage
But if SomePackage also exist on pypi pip will simply install the latest version amongst the indexes given (and not give any priority to my private registry). That seems like a risk to me and some others. So I was wondering if there could be a workaround to prevent the involuntary installation of SomePackage from pypi.
I have an idea, but maybe that's a bad one (or maybe it doesn't work as I expect):
# Install the package from the private registry without its dependencies:
pip install --no-deps --index-url http://my.package.repo/simple SomePackage
# Then install only its dependencies:
pip install --extra-index-url http://my.package.repo/simple SomePackage
This would only work for SomePackage but all its dependencies from the private registry will not be "protected" by this approach. Any other/better idea?

Related

Does pip only use PyPI or does it use other domains to find packages?

I know that by default, pip uses PyPI to look for packages. I would like to know if there are other domains other than PypI that pip uses.
PIP Can install from
PyPI
VCS project URL
Local project directories
Local or remote source archives
to run from a local passage you can input pip install /opt/mypackage
Finally, run pip install --help to see all installation options
PIP can install from many different sources. You can find the whole list here
You can also setup your own Python package repository and configure pip to install from there.

How to install a package with Poetry that requires CLI args?

Here's an example that requires it:
poetry add pycurl
... gives:
ImportError: pycurl: libcurl link-time ssl backend (openssl) is different from compile-time ssl backend (none/other)
The fix is given in that post:
pip install --compile --install-option="--with-openssl" pycurl
How to package-manage my project now?
Must I use poetry for everything else, and manually pip install pycurl?
Or can I somehow fold it into my pyproject.toml?
Although this is old, I would like to share an alternative to re-install the package with the CLI options.
I do not know if running this with Poetry could cause any damage, but I would believe not.
First you'd have to activate your poetry environment with poetry shell.
After that, get the current version of pycurl that is installed using pip list, and copy that value.
Now, you can run pip install --compile --install-option="--with-openssl" --upgrade --force-reinstall pycurl==<version> to install it with the correct options.

How to avoid pip install package again while conda install was done before?

guys:
I use conda install tensorflow-gputo install tensorflow 2.0 , and
numpy=1.20.2 would be one of the package installed, and then I use python3 -m pip install SOMEPACKAGE ,this SOMEPACKAGE needs numpy to be installed as well , but pip seems does not check or realize the package numpy has already installed...
I would like to show everything I know so far :
1.I know the packages installed via conda install would go to anaconda3/envs/YOUR_ENV/lib/site-packages
2.I use python3 -m pip install -t anaconda3/envs/YOUR_ENV/lib/site-packages to force the package would be installed to the place where conda install would be.
However,pip still tries to dwonload *.whl file and install package again,I do not want this package installation process happen again ,while it did mention that I can use --upgrade to replace the existed package...
So I would like to know
How does pip and conda install check if the target package has already existed before they actually to through install process?
I think using python3 you are not using interpreter from your current conda environment so it gets installed elsewhere
python -m pip install (or simply pip install) from your activated environment should work and ignore dependencies installed by conda if they satisfy the requirements

Installing dependencies for a pip-only package through conda

Sometimes I need to install a pip-only package into a conda environment. If I install the package using pip install, then all the dependencies for that package are installed using pip, even if they are available to conda.
I would like to install as many packages as possible through conda, so currently I use a hack to get the list of package dependencies through pip, search for all of them on conda, conda install the ones that are found, and then go through with the pip install.
Am I right to prefer installing dependencies through conda rather than pip? And if so, can anyone think of a more elegant way to solve this problem?
pip and conda are two separate package managers. Only in very rare cases package managers actually work together. In practical applications conda and pip usually do not.
In reality, mixing conda and pip packages is usually unavoidable. This often leads to a messy package management, as you describe.
In my opinion, the best and currently only proper way to solve this problem is to create a conda package for all (pypi-)packages and dependencies you want to use in your conda environments.
conda-forge is a community effort that offers an easy way to contribute your own package to the conda infrastructure. You may want to check out if your package is already available, and if not if contributing is an option for you.
Am I right to prefer installing dependencies through conda rather than pip?
Yes.
Anaconda has a blog post that discusses best practices when you have no choice but to combine conda and pip. Here's a list of guidelines from that blog post:
Best Practices Checklist
Use pip only after conda
install as many requirements as possible with conda, then use pip
pip should be run with --upgrade-strategy only-if-needed (the default)
Do not use pip with the --user argument, avoid all “users” installs
Use conda environments for isolation
create a conda environment to isolate any changes pip makes
environments take up little space thanks to hard links
care should be taken to avoid running pip in the “root” [base] environment
Recreate the environment if changes are needed
once pip has been used conda will be unaware of the changes
to install additional conda packages it is best to recreate the environment
Store conda and pip requirements in text files
package requirements can be passed to conda via the --file argument
pip accepts a list of Python packages with -r or --requirements
conda env will export or create environments based on a file with conda and pip requirements

Practical difference between pip and conda

I saw other questions regarding difference between pip and conda, but it is not clear to me yet, please consider that before marking as duplicate.
If I run pip install seaborn and conda install seaborn Will I get the same result ?
I can run pip install seaborn twice without any problem, but if I run pip install and then conda install do I get the same package duplicated in two different places ?
Conda and pip cannot be used interchangeably but what are examples of that ?
Both pip and conda install the package (pretty much) with the same end result. There may be minor differences, e.g. zipped egg or not, it depends a bit on how the conda package was created. The conda package is always a compiled binary distribution though, not a source distribution.
I don't think conda will install it in different places, it may well overwrite your pip package. But it's kind of risky because conda keeps nicely track of what's installed and figures out all dependencies betweeen all conda packages in the environment. You really want to limit yourself to conda packages and only install pip packages if you really have to. It's quite easy to create conda packages though from pip packages.
Not sure about "interchangeably", you can use them alongside each other. But pip and conda are not so aware of each other so you might run into trouble with say updating packages to new versions.
In summary: if you're using conda packages, best to stick with that. You get the best out of the conda ecosystem with it's package version and environment management.

Resources