Anaconda equivalent of "setup.py develop" - anaconda

How can I install a package under development to an Anaconda environment?
With pip:
pip install -e /path/to/mypackage
or with regular setuptools:
python /path/to/mypackage/setup.py develop

There is also conda develop available now.
http://conda.pydata.org/docs/commands/build/conda-develop.html
Update in 2019: conda develop hasn't been maintained and is not recommended. See https://github.com/conda/conda-build/issues/1992
Recommendation is to use python setup.py develop or pip install -e .

Using either of those will work with Anaconda. Make sure that you have pip or setuptools installed into the conda environment you want to install into, and that you have it activated.

This is the equivalent to pip install -e .
conda install conda-build
conda develop .

As explained in this gh issue thread, because of build isolation and dependency installation, Anaconda developers recommend using:
pip install --no-build-isolation --no-deps -e .

Build / Host Environment
To create build and host environments and a build script go to your recipe directory and use
conda debug /path/to/your/recipe-directory
as documented here. This will print an instructive message like
################################################################################
Build and/or host environments created for debugging. To enter a debugging environment:
cd /home/UserName/miniconda3/conda-bld/debug_1542385789430/work && source /home/UserName/miniconda3/conda-bld/debug_1542385789430/work/build_env_setup.sh
To run your build, you might want to start with running the conda_build.sh file.
################################################################################
(The message might tell you incorrectly, that it created a test environment.) Your source code has been copied to the .../work directory and there is also a conda_build.sh script. Note, that sourcing the build_env_setup.sh will load both build and host environments.
You can work on your code and your recipe and build with the conda_build.sh, but you won't get a proper conda package, as far as I know. When you are finished, you can remove the debug environment:
conda deactivate # maybe twice
conda build purge
Test Environment
To get the test environment, you have to build the package first and then debug that. This might be useful to fix your test files.
conda build /path/to/your/recipe-directory # creates mypackage*.tar.bz2
# find file location of mypackage*.tar.bz2 with:
conda search --info --use-local mypackage # look at the url row for the path
cd /path/to/miniconda3/conda-bld/linux-64/ # go to that path, can be different
conda debug mypackage*.tar.bz2
This will print e. g.:
################################################################################
Test environment created for debugging. To enter a debugging environment:
cd /home/UserName/miniconda3/conda-bld/debug_1542385789430/test_tmp && source /home/UserName/miniconda3/conda-bld/debug_1542385789430/work/conda_test_env_vars.sh
To run your tests, you might want to start with running the conda_test_runner.sh file.
################################################################################
Again, remove with
conda deactivate
conda build purge
Run Environment
This is actually no debugging, but the general process of building and installing a local package. With the run environment you can check, whether all dependencies are specified in the requirements/run section. Also pinning can be an issue.
(base) $ conda build /path/to/your/recipe-directory
(base) $ conda create --name package-env --use-local mypackage
(base) $ conda activate package-env
(package-env) $ python
>>> import mypackage
You can also list the dependencies of your package with (man page)
conda search --info --use-local mypackage
A last hint: If you want to know the versions of the dependencies and see, whether pinning works, try (man page)
conda render /path/to/your/recipe-directory

Related

Does poetry (or pip) 'install' create/have the option to deploy an executable?

I am running
poetry install
from within a python local virtualenv ".venv" . The project is supposed to create an executable hercl that becomes available on the user's path. Two questions:
What options / configuration of I'm not sure if that's supposed to gets installed into the local .venv/bin or in the pyenv shims.
Since poetry reuses / redirects many functions to pip it may be the case that the feature I'm asking about is actually from pip itself. I have not been able to discover from either poetry or pip documentation about this shell script installation. How is this achieved?
Update
After running running pip install outside of the virtualenv it pulls from pypi and creates a bash script ~/.pyenv/shims/my_app .
In my case the my_app is "hercl" and we see this:
$which hercl
~/.pyenv/shims/hercl
Its contents are :
$cat $(which hercl)
#!/usr/bin/env bash
set -e
[ -n "$PYENV_DEBUG" ] && set -x
program="${0##*/}"
export PYENV_ROOT="~/.pyenv"
exec "~/.pyenv/libexec/pyenv" exec "$program" "$#"
Somehow this script is installed when running pip install: I am wondering how pip knows to do this. Is it from the pyproject.ml from poetry ? Is it from a setup.py or setup.cfg associated with pip ?
Anoterh Update #sinoroc has another tack on this: poetry has a scripts section that I did not notice (noobie on that tool).
[tool.poetry.scripts]
hercl = "hercl.hercl:main"
hercl is a command that I was looking for .
But there was also an actual _bash script that would launch hercl that got installed under the shims as part of the virtualenv. i think that script were in the
In a Poetry-based project such executable scripts are defined in the scripts section of pyproject.toml.
If a virtual environment is active when installing the application then the executable is installed in the virtual environment's bin directory. So it is available only while the virtual environment is "active".

The environment is inconsistent, please check the package plan carefully

I tried to update or install new packages from anaconda and lately, this message has appeared:
The environment is inconsistent, please check the package plan carefully
The following package are causing the inconsistency:
- defaults/win-32::anaconda==5.3.1=py37_0
done
I tried with conda clean --all and then conda update --all but it persists.
Conda Info
active environment : base
active env location : C:\Users\NAME\Continuum
shell level : 1
user config file : C:\Users\NAME\.condarc
populated config files : C:\Users\NAME\.condarc
conda version : 4.6.11
conda-build version : 3.17.7
python version : 3.7.3.final.0
base environment : C:\Users\NAME\Continuum (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/win-32
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/win-32
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/win-32
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/msys2/win-32
https://repo.anaconda.com/pkgs/msys2/noarch
package cache : C:\Users\NAME\Continuum\pkgs
C:\Users\NAME\.conda\pkgs
C:\Users\NAME\AppData\Local\conda\conda\pkgs
envs directories : C:\Users\NAME\Continuum\envs
C:\Users\NAME\.conda\envs
C:\Users\NAME\AppData\Local\conda\conda\envs
platform : win-32
user-agent : conda/4.6.11 requests/2.21.0 CPython/3.7.3 Windows/10 Windows/10.0.17763
administrator : False
netrc file : None
offline mode : False
I had faced the same problem. Simply running
conda install anaconda
solved the problem for me.
saw this on Google Groups
This message was added in conda 4.6.9, previously there was no indication when conda detected an inconsistent environment unless conda was run in debug mode. It is likely that your environment was inconsistent for some time but the upgrade to conda made it visible. The best option it to run "conda install package_name" for the inconsistent packages to let conda try to restore consistency.
and it really works for me.
Maybe you should try conda install anaconda in your situation.
The inconsistencies are caused due to different versions of the packages, and their clashing dependencies.
conda update --all
This command updates all the packages, and then conda solves the inconsistency on its own.
Had this same problem and none of the other solutions worked for me. Ended up having to uninstall and reinstall conda, then reinstall all of my libraries.
Ultimate solutions:
conda activate base
conda install anaconda
conda update --all
Works on Windows 10 and Ubuntu 18.04 (credits to #MF.OX for ubuntu).
Removed following problems for me:
The environment is inconsistent
WARNING conda.base.context:use_only_tar_bz2(632)
If the other solutions don't work, reverting the environment can fix this.
Use conda list --revisions, pick a revision number, and use conda install --revision [#] going back step-by-step until everything works again.
Given a situation like the following,
> conda update -c intel --all
Collecting package metadata: done
Solving environment: |
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- intel/win-64::ipython==6.3.1=py36_3
- intel/win-64::prompt_toolkit==1.0.15=py36_2
done
As mentioned in other answers, the idea is to have some sort of re-installation to occur for the inconsistent packages.
Thus, with a few copy-&-paste's, you could:
> conda install intel/win-64::ipython==6.3.1=py36_3
Collecting package metadata: done
Solving environment: /
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- intel/win-64::ipython==6.3.1=py36_3
- intel/win-64::prompt_toolkit==1.0.15=py36_2
done
## Package Plan ##
environment location: c:\conda
added / updated specs:
- ipython
The following NEW packages will be INSTALLED:
jedi intel/win-64::jedi-0.12.0-py36_2
parso intel/win-64::parso-0.2.0-py36_2
pygments intel/win-64::pygments-2.2.0-py36_5
wcwidth intel/win-64::wcwidth-0.1.7-py36_6
Proceed ([y]/n)? y
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(and you would have to repeat for all the packages)
My “Shortcut”
Alternatively, cook up an (ugly) one-liner (this should work for Windows as well as other platforms)
Note: by "ORIGINAL_COMMAND", I'm referring to any command that gives you the error message (without any other side-effects, ideally)
<ORIGINAL_COMMAND> 2>&1 | python -c "import sys,re,conda.cli; conda.cli.main('conda','install','-y',*re.findall(r'^\s*-\s*(\S+)$',sys.stdin.read(),re.MULTILINE))"
Expanding the above one-liner:
from re import findall, MULTILINE
from sys import stdin
from conda.cli import main
main(
"conda", "install", "-y",
"--force", # Maybe add a '--force'/'--force-reinstall' (I didn't add it for the one-liner above)
*findall(r"^\s*-\s*(\S+)$", stdin.read(), MULTILINE) # Here are the offenders
)
I was getting an environment is inconsistent error when I tried to update my base conda environment. I'm using miniconda. Unfortunately, none of the answers above worked for me.
What did work for me was:
conda activate base
conda install conda --force-reinstall
conda install conda --force-reinstall
conda update --all
(Yes, for some reason it was necessary to run conda install conda --force-reinstall twice!)
The command conda install -c anaconda anaconda did the trick for me. For my setup, I need to specify the channel otherwise it would not work. After running the command in the terminal, I was prompted to update a list of packages that was found to be inconsistent. Without this step, I was not able to install or update any packages with conda install <package_name> or conda update <package_name respectively.
What worked for me was to
`conda remove <offending_packagename>`,
`conda update --all`
and then finally
`conda install <offending_packagename>`.
I had this problem for ages. The conda install anaconda might work, but it takes just way too long -- more than 24 hours on my machine.
Here is a solution that worked for me in under 5 minutes:
Remove all the unneeded packages -- being careful to leave the ones that are essential for conda to operate.
Then, use conda install anaconda.
But how?? there is a lot of them!
This is what I have done:
Make a fresh envinroment with python, fairly bare-bone. then, list the packages in there:
conda create -n fresh python
conda activate fresh
conda list
Save the output, you will need it.
1b. go back to the base envinroment:
conda deactivate
use the following snippet to generate a conda command that will remove all the inconsistent packages:
(good packages are)
exclusion_text = '''
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
anyio 2.2.0 py39h06a4308_1
argon2-cffi 20.1.0 py39h27cfd23_1
async_generator 1.10 pyhd3eb1b0_0
...
... and more! get this from a good environment.
Note the usage of triple quotes (''') to use a multiline-string in python.
bad_packages_text = '''
- https://repo.continuum.io/pkgs/main/linux-64/networkx-2.1-py36_0.tar.bz2/linux-64::networkx==2.1=py36_0
- https://repo.continuum.io/pkgs/main/linux-64/spyder-3.2.6-> py36_0.tar.bz2/linux-64::spyder==3.2.6=py36_0
py36h4c697fb_0.tar.bz2/linux-64::jdcal==1.3=py36h4c697fb_0
- defaults/noarch::jupyterlab_server==1.1.4=py_0
- defaults/linux-64::argh==0.26.2=py37_0
...
... and more! get this by copy-pasting the "The following packages are causing the inconsistency." message.
then, in python, process this:
exclusions = [line.split(' ')[0] for line in exclusion_text_lines if line !='']
bad_packages_lines = bad_packages_text.split('\n')
bad_packages = [line.split('::')[1].split('==')[0] for line in bad_packages_lines if line!='']
exclusions.append('conda') # make sure!
exclusions.append('tqdm')
finally, construct the life-saving command:
command_line = 'conda remove '
for bad_package in bad_packages:
if bad_package not in exclusions:
command_line = f'{command_line} {bad_package}'
command_line
Since in solving the environment, all the packages on the remove list can be ignored, conda no longer needs to consider their versions, and the process is fast.
Possibly someone can refactor this method to make it easier -- or better yet, upgrade conda to enable quick reset base command.
This worked for me -- it took me longer to write this post than to execute these actions.
Good luck!
To those of us who have miniconda and can't/don't want to install anaconda: the accepted answer works when adapted.
conda install conda
conda update --all
Would have commented, but my rep is too low.
conda install anaconda
conda clean --all
conda update --all
fix the problem for me
To solve this message I had to run conda update --all in my base environment three times after each other.
Every time the number of inconsistent packages decreased until conda said:
# All requested packages already installed.
I'm on macOS Big Sur 11.6 using conda version 4.10.3.
In my case, none of the above worked. But this did the trick in less than a minute:
1- I downloaded again the lastest installer (miniconda in my case)
2- Run the installer with the -u option:
bash Miniconda3-py39_xxxx-Linux-x86_64.sh -u
3- Answer yes to all questions and let the installer finish
4- Then I could run conda update conda -all
Hope this helps...
You probably installed anaconda with python 2.7 but later you used python 3.x. Thus, you are getting an error message. In my case, I solved the problem by activating anaconda with python 2.7:
conda create --name py2 python=2.7
Try to have a look to the environment management
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html
By using something along the lines
conda create --name astra python=3.5
conda activate astra
conda install -c astra-toolbox astra-toolbox
You can see that you can even specify target python version. Now play with the new packages installed. When unsatisfied, you can always do
conda deactivate
conda env remove -n astra
If you install everything to the base env and something gets broken, then probably better is not to install conda at all and go with default python managing it through pip.
In my environment.
1.
conda install anaconda
conda update --all
Then it works correctly.

Import System Modules in a Conda Environment

I have installed caffe and pytorch0.3 in my system path(environment), and there is a project which only works under caffe and history version of pytorch0.2. To solve that I install pytorch0.2 in a Conda environment and I wonder if there is any way can save me from installing a caffe again in this conda environment.
In other words, can I use pytorch0.2 in this Conda environment meanwhile import the system caffe? and how?
Activate your environment and try the following
conda install package-name --offline
Also, in case you wish to clone the root or some other environment to a conda environment, you can use -- clone. For instance, when you wish to clone the root -
conda create -n pytorch_02 --clone root

conda equivalent of pip install --user

To install to my own directory I can use
pip install --user package
Alternatively I can use
conda install package
How do I ask conda to install to my home directory since conda does not take a --user flag?
Installing pip packages to $HOME folder
I don't think it's possible. Use virtual environments (conda create).
See -p option here:
-p PATH, --prefix PATH
Full path to environment prefix.
So to install to, say, local under your home directory, do:
conda install -p $HOME/local packagename
Note, however, this is not a "clean" install as it adds a bunch of conda-related files.
To install conda packages on your own directory you can follow these steps:
Create a blank environment
conda create -y -n my-conda-env
Replace the name my-conda-env with any name you want to give the environment.
Activate the environment
source activate my-conda-env
Don't forget to replace my-conda-env with the name you gave the conda environment from the previous step
Install your package
conda install -c bioconda epa-ng
And that's it, the package should be installed on your own directory
Simply:
sudo conda install -c conda-forge package
Or:
sudo chmod -R 777 ./
conda install -c conda-forge package
I don't know of an exact match for the --user flag, but a reasonable analogue is to use a virtual environment.
What I do when I have to install to a shared CentOS server where I don't have admin access:
First I run
conda env list
will list all conda virtual environments and display the path to each. Once you have the environment created and can see it in the conda env list, copy the path to the environment.
If you need to create one, you can do that with conda create or by running anaconda-navigator and using the GUI.
Activate your environment (if not active) with
conda activate [environment_name]
or
activate [environment_name]
depending on your system (most linux systems use the first, Windows and CentOS use the latter).
Now you can use
conda install -p [environment_path] [package_name]
and you are off to the races.
This is really a work around; it's not the best but it does install the package to the selected virtual environment.
The current Anaconda Install Individual Edition, when run in a linux local account, installs in a local directory. So all the subsequent installs should install there, too.
According to the documentation:
--use-local

Weird behavior of conda when creating empty environments

I create a conda environment without specifying any packages using the following command:
conda create --name test_env
I can then use all the packages in the root environment inside test_env (but they do not appear in the outputs of conda list and conda env export). This is already unexpected to me but the real problems begin when I then install something inside that environment, e.g.:
conda install pywavelets
Afterwards, pywavelets is usable but all the other packages which are no dependencies of pywavelets disappear inside the environment (e.g. pandas). I don't understand why that happens. Does anybody have an explanation for that?
More importantly, what does this mean for best practices for working with conda environments? Should I always create my environments specifying at least python (conda create --name test_env python)? However, then I have to install everything by hand in that environment which is quite cumbersome. So, my idea now is to specify anaconda for all environments I create:
conda create --name test_env anaconda
The disadvantage, however, is that the list of dependencies listed by conda list and conda env export gets unnecessarily long (e.g. even listing the Anaconda Navigator). Does anybody have a better solution for this?
The reason you can use all the packages from the root environment when you don't specify a Python version during environment creation is because you're actually using the root environment's Python executable! You can check with which python or python -c "import sys; print(sys.executable)". See also my other answer here.
When you install pywavelets, one of the dependencies is (probably) Python, so a new Python executable is installed into your environment. Therefore, when you run Python, it only picks up the packages that are installed in the test_env.
If you want all of the packages from another environment, you can create a file that lists all the packages and then use that file to create a new environment, as detailed in the Conda docs: https://conda.io/docs/user-guide/tasks/manage-environments.html#building-identical-conda-environments
To summarize
conda list --explicit > spec-file.txt
conda create --name myenv --file spec-file.txt
or to install into an existing environment
conda install --name myenv --file spec-file.txt
Since that's just a text file, you can edit and remove any packages that you don't want.

Resources