I'm new to conda. Is there a way I can build the package somewhere, move it to another environment and install it without connect to the internet?
For example we want to have a package specifically for xgboost users. We want have these in the meta.yaml:
package:
name: xgboost-test
version: 1.0
build:
skip_compile_pyc:
- "*"
# needs channels: conda-forge
requirements:
host:
- python
- pip
- conda
run:
- python
- pip
- git
- conda
- nb_conda
- pandas
- xgboost==1.1.1
After building a new package which contains all above, we have a new package named xgboost-test-1.0.20221206-py37_0.tar.bz2.
Is there any way I can only move this new package to another computer and install it over there, without setting the channel, or not connect to the internew? I don't think it's convenient to setup a channel every time. I thought there is a way which I can simply install the new package file, that's more quick and straight forward.
The best way I know is copy the whole channel folder to the new environment and install over there. The other way I know is to download the package file, use conda index to initialize the channel, and install it by conda install -c file:///path/to/local/channel.
But I'm wondering if there is a way just move the package itself, not the whole channel folder. I thought there is a way which I can simply install the new package file, that's more quick and straight forward.
Related
I recently developed a package my_package and am hosting it on GitHub. For easy installation and use, I have following setup.py:
from setuptools import setup
setup(name='my_package',
version='1.0',
description='My super cool package',
url='https://github.com/my_name/my_package',
packages=['my_package'],
python_requieres='3.9',
install_requires=[
'some_package==1.0.0'
])
Now I am trying to install this package in a conda environment:
conda create --name myenv python=3.9
conda activate myenv
pip install git+'https://github.com/my_name/my_package'
So far so good. If I try to use it in the project folder, everything works perfectly. If I try to use the packet outside the project folder (still inside the conda environment), I get the following error:
ModuleNotFoundError: No module named 'my_package'
I am working on windows, if that matters.
EDIT:
I'm verifying that both python and pip are pointing towards the correct version with:
which pip
which python
/c/Anaconda3/envs/my_env/python
/c/Anaconda3/envs/my_env/Scripts/pip
Also, when I run:
pip show my_package
I get a description of my package. So pip finds it, but as soon as I try to import my_package in the script, I get the described error.
I also verified that the package is installed in my environment. So in /c/Anaconda3/envs/my_env/lib/site-packages there is a folder my_package-1.0.dist-info/
Further: python "import sys, print(sys.path)"
shows, among other paths, /c/Anaconda3/envs/my_env/lib/site-packages. So it is in the path.
Check if you are using some explicit shebang in your script pointing to other Python interpreters.
Eg. using the system default Python:
#!/bin/env python
...
While inside your environment myenv, try to uninstall your package first, to do a clean test:
pip uninstall my_package
Also, you have a typo in your setup.py: python_requieres --> python_requires.
And I actually tried to install with your setup.py, and also got ModuleNotFoundError - but because it didn't properly install due to install_requires:
ERROR: Could not find a version that satisfies the requirement some_package==1.0.0
So, check also that everything installs without errors and warnings.
Hope that helps.
First thing I would like to point out (not the solution) regards the following statement you made:
If I try to use it in the project folder [...] If I try to use the packet outside the project folder [...]
I understand "project folder" means the "my_package" folder (inside the git repository). If that is the case, I would like to point out that you are mixing two situations: that of testing a (remote) package installation, while in your (local) repository. Which is not necessarily wrong, but error-prone.
Whenever testing the setup/install process of a package, make sure to move far from your repository (say, "/tmp/" equivalent in Windows) and, preferably, use a fresh environment. That will eliminate "noise" in your tests.
First thing I would tell you to do -- if not already -- is to create a fresh conda env and install your package from an empty/new folder. Eg,
$ conda env create -n test_my_package ipython pip
$ cd /tmp # equivalent temporary or new in your Windows
$ pip install git+https://github.com/my_name/my_package
If that doesn't work (maybe a problem with your pip' git+http code), do another way: create a release for your package (eg, "v1") and then install the released version by indicating the zip package URL (that you get from your "my_package" releases page on Github):
$ pip install https://github.com/my_name/my_package/archive/v1.zip
I'd like to distribute multiple versions of a package through conda. Specifically, I'd like to do something like this:
...
package-v1.2-dev
package-v1.2
package-v1.1-dev
package-v1.1
package-v1.0
The trick is that I'd like to have the "latest" or default package be the release versions that do not have -dev. As I understand it, conda install <package> without a version number will install the newest build. In my case, that will always be -dev. Is it possible to make the default a specific version number?
You can achieve this by specifying a custom "label" for your dev packages. Keep using the default main label for your release packages, but use a non-main label (e.g. dev) for the other packages.
First, a quick note about version numbers: conda package versions must not contain the - character, so v1.2-dev is not a valid version. For the following examples, I'll use v1.2.dev.
Here's how to upload your packages:
anaconda upload mypackage-v1.2.tar.bz2
anaconda upload --label dev mypackage-v1.2.dev.tar.bz2
(You can also manipulate the labels for existing packages via your account on the http://anaconda.org website.)
By default, your users will only download your main packages. Users who want the dev packages will have two choices:
They can specify the dev label on the command-line:
conda install -c mychannel/label/dev mypackage
OR
They can add your dev label to their .condarc config
# .condarc
channels:
- mychannel/label/dev # dev label
- mychannel # main label only
- conda-forge
- defaults
And then there's no need to specify the channel on the command-line:
conda install mypackage
PS -- Here's a side note about something you wrote above:
As I understand it, conda install <package> without a version number will install the newest build
Just to clarify, it doesn't install the "newest" in chronological sense, but rather the highest compatible version according to conda's VersionOrder logic. That logic is designed to be largely compatible with relevant Python conventions (e.g. PEP440 and others), but with some affordances for compatibility with other languages' conventions, too.
Please note: As far as conda (and PEP440) is concerned, 1.2.dev comes BEFORE 1.2. (Maybe you already knew that, but I don't consider it obvious.)
$ python
>>> from conda.models.version import VersionOrder
>>> VersionOrder('1.2.dev') < VersionOrder('1.2')
True
I have used anaconda3 for a few projects recently, and every time I create a virtual environment for a project, it seems that anaconda is re-downloading the same packages (pytorch, for instance).
Have I misconfigured something or this behavior is OK?
for clerification, I am doing the Stanford CS224n course and for the assignments I use:
conda env create --file env.yml
Where env.yml is of the form:
name: local_nmt
channels:
- pytorch
- defaults
dependencies:
- python=3.5
- numpy
- scipy
- tqdm
- docopt
- pytorch
- nltk
- torchvision
I couldn't fine an explanation in the anaconda documentation.
Thanks in advance!
If only the package name or version is specified, then Conda will default to grabbing the latest versions that are consistent with constraints. Hence, any packages that have newer builds available will result in downloading.
Offline Mode
There is an --offline flag to only use what is available in the package cache.
Specifying Builds
However, that may not always be feasible (e.g., you've added some non-cached packages to the YAML). In that case, one could additionally specify the build (which sort of serves as a unique identifier) to correspond to the already cached versions.
Not sure the cleanest way to do that, but one approach would be to first export a YAML from your existing environments where the packages exist (e.g., conda export env > env.yaml), and then use the specifications in there to fill in the details for the environment YAML you are trying to create.
Cloning
It is likely also worth mentioning that one can also clone existing environments:
conda create --clone old_env --name new_env
I'm having issues with pip failing to install editable packages from a local directory. I was able to install the packages manually using commands like pip install -e pkg1. I wanted to use a requirements.txt file to automate future installs, because my coworkers will be working on the same packages. My ideal development workflow is for each developer to checkout the source from version control and run pip install -r requirements.txt. The requirements file would designate all the packages as editable so we can import our code without the need for .pth files but we wouldn't have to keep updating our environments. And by using namespace packages, we can decouple the import semantics from the file structures.
But it's not working out.
I have a directory with packages like so:
index/
pkg1/
src/
pkg1/
__init__.py
pkg1.py
setup.py
pkg2/
src/
...etc.
Each setup.py file contains something like:
from setuptools import setup, find_packages
setup(
name="pkg1",
version="0.1",
packages=find_packages('src'),
package_dir={'':'src'},
)
I generated my requirements.txt file using pip freeze, which yielded something like this:
# Editable install with no version control (pkg1==0.1)
-e c:\source\pkg1
# Editable install with no version control (pkg2==0.1)
-e c:\source\pkg2
...etc...
I was surprised when pip choked on the requirements file that it created for itself:
(venv) C:\Source>pip install -r requirements.txt
c:sourcepkg1 should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+
Also, some of our packages rely on other of our packages and pip has been absolutely useless at identifying these dependencies. I have resorted to manually installing packages in dependency order.
Maybe I'm pushing pip to its limits here. The documentation and help online has not been helpful, so far. Most sources discuss editable installation, installation from requirements files, package dependencies, or namespace packages, but never all these concepts at once. Usually when the online help is scarce, it means that I'm trying to use a tool for something it wasn't intended to do or I've discovered a bug.
Is this development process viable? Do I need to make a private package index or something?
Jython Package installation issue, using pip
Hi, I have installed Jython2.7 configured with pydev in eclipse neon, also configured python 3.6 package
I am able to install packages for python using pip installer?
pip install "packagename"
Below are some of the packages in python/Lib/Site-packages directory
I was able to install all the packages
How do I use pip installer to install packages for jython?
I tried to install Jip package with
jython install setup.py
The binary File got installed in the Jython/Lib/Site-packages folder
However, I am not able to use it.
where and how do I get Jython package binaries like jip?
Also, Please let me know how to search jython packages?
Also, How to make pip install library packages in jython?
Any other configuration like jython home, etc that should be made?
This answer is going to be really generic but I just recently have slogged my way through the setup for jython/jip/pip and here's roughly what I had to do.
Firstly, I'm running Windows 7 64 Bit from behind a proxy (work machine.)
Had to install jython 2.7.0 instead of 2.7.1 because (I think anyway) 2.7.1 requires admin privileges which I don't have on my work PC.
Pip didn't install correctly during the Jython installation and I spent an obscene amount of time trying to get it installed and functioning as I knew it from my cpython days. NOTE: Just because you get pip installed, doesn't mean you can use any package on a python package repo. As of 2.7.0, Jython doesn't have end to end capability to interpret/compile some libraries that rely on certain python wrappers of native OS function calls. I believe 2.7.1 makes solid progress in the direction of supporting all needed native calls but don't quote me on that. For example, I tried to use wxPython to make a simple GUI to test my jython install. Trying to install it from pip kept causing really non-specific error info that took me a lot of time to figure out that the cause was jython simply couldn't compile the wxPython source so beware.
I had to set environment variables 'http_proxy' and 'https_proxy' in the form of http://proxyhosturl:port and https://proxyhosturl:port respectively to get out from behind the proxies without having to invoke pip with the proxy switch every time I called it.
To actually install pip, have a look here. These instructions are for Python and Linux/Unix but the principle is roughly the same. Just use jython -m instead of python -m and ignore the '$' at the start of each command line.
Also be sure to CD to your python_home/bin folder when invoking the ez_install exe.
If that doesn't work (didn't for me), try using get-pip.py script with these instructions https://pip.pypa.io/en/stable/installing/ (remember jython instead of python etc.). Download it, cd to the download location and follow the noted install steps. Worth noting is about half way down the install instructions where it details installing from local archives (source/binary zip or tar.gz archives of pip and setuptools as better described here: https://packaging.python.org/tutorials/installing-packages/#installing-from-local-archives).
The links to the bin archives of pip and setuptools are here:
https://pypi.python.org/pypi/setuptools
https://pypi.python.org/pypi/pip
It may also be worth making sure that your PATH environment variable has the jython/bin path in the variable value. The jython installer should do this but, again, mine did not.
If all goes well, you should be able to invoke pip with the --version switch and if it prints a line with the installed pip version info then you should be good to go
Another quirky issue I had was I could invoke a function of pip one time and any subsequent times I would get a stack trace ending with something along the lines of an object not having a certain property. I fixed that by finding my temp directory by opening a windows explorer instance and typing %TEMP% in the address bar and hitting enter, it should take you to a subdir of your AppData folder and there you may see a folder with the name of the package you were trying to install and the text "_pip" somewhere in the directory name. Delete the directory and try the pip install command again. I had to do this + invoke pip install pip -U to update my install to the latest version. Then pip began behaving correctly in my instance.
pip search numpy (or your library name) will generate a list of results with the same logic it uses to locate your desired package when you call pip install but, again, just because it returns a matching package doesn't mean it will compile when you install it (numpy doesn't work because of the missing java to C native function calls I described earlier.) The trade off is that you can import code artifacts from Java JAR files in your Jython script files and leverage their functionality with relative ease. Between the public Java APIs available and the python packages that work with the jython interpretor, you can (in my experience) come up with a way to accomplish your task. See the following info on JIP, Maven and IDEs.
IDE and jython integration (Eclipse)
- If you are stuck using Eclipse (like me) it actually has pretty decent support for python development. Install the PyDev plugin for Eclipse from Help -> Install Software. Put in this URL https://marketplace.eclipse.org/content/pydev-python-ide-eclipse, hit tab, and select the PyDev plugin and hit 'finish.'
- Setup the jython interpretor info from Windows -> Preferences -> PyDev. Provide the path to your jython.jar file.
- You should now be able to use File -> New PyDev project to create a basic python project and configure it to use your version of Jython and Java.
Brief Overview of Jip and Maven
- jip is a jython package that is invoked very similarly to pip but instead will download JAR files from the Maven Central Repository instead of python packages from pypi.com, for example. See the install instructions described here. Note the install procedure for a global jip install which differ from just pip install jip. https://pypi.python.org/pypi/jip/
- I never got jip to work exactly as I wished because there's not a ton of documentation on it outside of what I already linked. However, if you install a JAR using jip, you have to go to your project in Eclipse and actually add the JARs themselves to your PYTHONPATH in order for import statements and editing to have intellisense and so that you don't get a classnotfound exception at runtime. See following screen shot.
- There is a JIP config file that you can use similar to the pip config ini file but I have yet to find any exhaustive documentation on it's setup.
Note in the above screen shot the first entry in the External libraries entries. By default, pip places installed packages in that directory so to enable eclipse to find them, you need to also ensure that location is entered.
In Conclusion
- I have more to add to this answer and I will do so as soon as possible. In the meantime, see this example project I've loaded into github.
https://github.com/jheidlage1222/jython_java_integration_example
It shows basic config and how to interface with JARs from python code. I used the apache httpcomponents library as an example. Good luck amigo.