I have several different environments in anaconda. When I try to upgrade all those environments via conda update --all after use "source .../activate [env ]" it downloadeds all packages for each environment separately.
However, most packages are exactly the same... is there any way I can avoid downloading the same packages for each environment?
When updating or installing a package with conda it will check if that exact package already exists in the ~/anaconda/pkgs directory. If the exact package is present conda will link this package rather than downloading it again. You should see the list of packages being downloaded in the output:
The following packages will be downloaded:
package | build
---------------------------|-----------------
and what is being installed to the environment:
The following packages will be UPDATED:
Packages listed under the "UPDATED" section and not under the "downloaded" section are being linked from the ~/anaconda/pkgs/ directory.
Related
I recently developed a package my_package and am hosting it on GitHub. For easy installation and use, I have following setup.py:
from setuptools import setup
setup(name='my_package',
version='1.0',
description='My super cool package',
url='https://github.com/my_name/my_package',
packages=['my_package'],
python_requieres='3.9',
install_requires=[
'some_package==1.0.0'
])
Now I am trying to install this package in a conda environment:
conda create --name myenv python=3.9
conda activate myenv
pip install git+'https://github.com/my_name/my_package'
So far so good. If I try to use it in the project folder, everything works perfectly. If I try to use the packet outside the project folder (still inside the conda environment), I get the following error:
ModuleNotFoundError: No module named 'my_package'
I am working on windows, if that matters.
EDIT:
I'm verifying that both python and pip are pointing towards the correct version with:
which pip
which python
/c/Anaconda3/envs/my_env/python
/c/Anaconda3/envs/my_env/Scripts/pip
Also, when I run:
pip show my_package
I get a description of my package. So pip finds it, but as soon as I try to import my_package in the script, I get the described error.
I also verified that the package is installed in my environment. So in /c/Anaconda3/envs/my_env/lib/site-packages there is a folder my_package-1.0.dist-info/
Further: python "import sys, print(sys.path)"
shows, among other paths, /c/Anaconda3/envs/my_env/lib/site-packages. So it is in the path.
Check if you are using some explicit shebang in your script pointing to other Python interpreters.
Eg. using the system default Python:
#!/bin/env python
...
While inside your environment myenv, try to uninstall your package first, to do a clean test:
pip uninstall my_package
Also, you have a typo in your setup.py: python_requieres --> python_requires.
And I actually tried to install with your setup.py, and also got ModuleNotFoundError - but because it didn't properly install due to install_requires:
ERROR: Could not find a version that satisfies the requirement some_package==1.0.0
So, check also that everything installs without errors and warnings.
Hope that helps.
First thing I would like to point out (not the solution) regards the following statement you made:
If I try to use it in the project folder [...] If I try to use the packet outside the project folder [...]
I understand "project folder" means the "my_package" folder (inside the git repository). If that is the case, I would like to point out that you are mixing two situations: that of testing a (remote) package installation, while in your (local) repository. Which is not necessarily wrong, but error-prone.
Whenever testing the setup/install process of a package, make sure to move far from your repository (say, "/tmp/" equivalent in Windows) and, preferably, use a fresh environment. That will eliminate "noise" in your tests.
First thing I would tell you to do -- if not already -- is to create a fresh conda env and install your package from an empty/new folder. Eg,
$ conda env create -n test_my_package ipython pip
$ cd /tmp # equivalent temporary or new in your Windows
$ pip install git+https://github.com/my_name/my_package
If that doesn't work (maybe a problem with your pip' git+http code), do another way: create a release for your package (eg, "v1") and then install the released version by indicating the zip package URL (that you get from your "my_package" releases page on Github):
$ pip install https://github.com/my_name/my_package/archive/v1.zip
I'm having issues with pip failing to install editable packages from a local directory. I was able to install the packages manually using commands like pip install -e pkg1. I wanted to use a requirements.txt file to automate future installs, because my coworkers will be working on the same packages. My ideal development workflow is for each developer to checkout the source from version control and run pip install -r requirements.txt. The requirements file would designate all the packages as editable so we can import our code without the need for .pth files but we wouldn't have to keep updating our environments. And by using namespace packages, we can decouple the import semantics from the file structures.
But it's not working out.
I have a directory with packages like so:
index/
pkg1/
src/
pkg1/
__init__.py
pkg1.py
setup.py
pkg2/
src/
...etc.
Each setup.py file contains something like:
from setuptools import setup, find_packages
setup(
name="pkg1",
version="0.1",
packages=find_packages('src'),
package_dir={'':'src'},
)
I generated my requirements.txt file using pip freeze, which yielded something like this:
# Editable install with no version control (pkg1==0.1)
-e c:\source\pkg1
# Editable install with no version control (pkg2==0.1)
-e c:\source\pkg2
...etc...
I was surprised when pip choked on the requirements file that it created for itself:
(venv) C:\Source>pip install -r requirements.txt
c:sourcepkg1 should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+
Also, some of our packages rely on other of our packages and pip has been absolutely useless at identifying these dependencies. I have resorted to manually installing packages in dependency order.
Maybe I'm pushing pip to its limits here. The documentation and help online has not been helpful, so far. Most sources discuss editable installation, installation from requirements files, package dependencies, or namespace packages, but never all these concepts at once. Usually when the online help is scarce, it means that I'm trying to use a tool for something it wasn't intended to do or I've discovered a bug.
Is this development process viable? Do I need to make a private package index or something?
I want to be able to pip install a package that installs a dependency package from GitHub. I want the version of that dependency package it installs to be the latest code in the master branch of the repository (i.e. I am not referencing a release of the package) (and there is a different version of the package for Python 2 and for Python 3). When I attempt to do this, the dependency is ignored. How can I get the dependency to be picked up and installed?
In setup.py I have lines like the following:
dependency_links = [
"git+https://github.com/veox/python2-krakenex.git;python_version<'3.0'",
"git+https://github.com/veox/python3-krakenex.git;python_version>='3.0'",
],
When I run pip, I do it using commands of the following form:
sudo pip install package_name --upgrade --process-dependency-links
I don't think it's possible. dependency_links aren't versioned, they're simple a list of URLs for packages listed in install_requires; those packages could be versioned but not in your case — you're trying to provide 2 URLs for one package which would confuse pip.
Perhaps you could rename one of the packages and provide package names
in the URLs:
install_requires=[
'krakenex;python_version<3',
'krakenex3;python_version>=3',
],
dependency_links = [
"git+https://github.com/veox/python2-krakenex.git#egg=krakenex;python_version<'3.0'",
"git+https://github.com/veox/python3-krakenex.git#egg=krakenex3;python_version>='3.0'",
],
I want to install the package dataframe of Octave on one of my servers, which does not have internet access. I used my laptop to download dataframe-1.1.0.tar.gz. I wonder how I can install it on my server manually.
In the README.html of Octave 4.0.0 folder you can find the following passage:
Included Octave Forge Packages
A number of Octave-Forge packages have been included with Octave, however they must be installed in order to use them.
To install:
• Start Octave and then open the build_packages.m file found in the src folder where Octave was installed.
• Run the script build_packages.m to build and install the packages.
Installation is a one-time procedure. After installation packages must still be loaded in order to use them with the pkg load PACKAGENAME command.
Other packages are available from Octave-Forge.
What you need to do for other packages, which are not included with Octave, is: download the package from http://octave.sourceforge.net/packages.php. Then put the package in the src folder and modify build_packages.m respectively before executing it.
According to the Octave documentation:
37.1 Installing and Removing Packages
Assuming a package is available in the file image-1.0.0.tar.gz it can
be installed from the Octave prompt with the command
pkg install image-1.0.0.tar.gz
If the package is installed successfully nothing will be printed on
the prompt, but if an error occurred during installation it will be
reported. It is possible to install several packages at once by
writing several package files after the pkg install command. If a
different version of the package is already installed it will be
removed prior to installing the new package. This makes it easy to
upgrade and downgrade the version of a package, but makes it
impossible to have several versions of the same package installed at
once.
I'm using Haskell Platform 7.10.2-a (64-bit) on Windows:
>cabal -V
cabal-install version 1.22.6.0
using version 1.22.4.0 of the Cabal library
My proxy requires (basic HTTP) authentication:
>set http_proxy=http://user:passwd#acme.com:port
It seems to work for cabal update:
> cabal update
Downloading the latest package list from hackage.haskell.org
Skipping download: Local and remote files match.
However, when I try to install any package, it fails:
> cabal get ghc-mod
Warning: The package list for 'hackage.haskell.org' does not exist. Run 'cabal
update' to download it.
cabal: There is no package named 'ghc-mod'.
>cabal install shelltestrunner
Warning: The package list for 'hackage.haskell.org' does not exist. Run 'cabal
update' to download it.
cabal: There is no package named 'shelltestrunner'.
You may need to run 'cabal update' to get the latest list of available
packages.
How can I get get or install to actually use the proxy? Or is there some other problem preventing installation of packages from hackage?
I may have found the cause, when using cabal -v update, it displays path like
\\ACME.NET\UserData\username\RF\AppData\Roaming\cabal\packages\hackage.haskell.org\00-index.tar.gz
while other uses of cabal use path like C:\Users\username\AppData\.., but without the RF folder in the displayed path. Must be my company network server config. Anyway, I updated paths like remote-repo-cache to use another folder in cabal/config and it works better now.