I am trying to create a conda environment with some required packages which are not in the official channels:
create --name stat_learn --file requirements.txt
However it fails with the following error:
PackagesNotFoundError: The following packages are not available from current channels:
I am using ubuntu 18.04, but I don't think it is related to my system. Here you can find a sample requirements file to reproduce the error:
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: linux-64
r-plotroc=2.2.0=r34hd3e613b_0
Is there a way to either i) produce a requirements file with channels included that would make the creation automatic or ii) let conda search itself for unofficial channels and find the required packages?
Thank you in advance, hope the question is clear :)
Related
I cannot easily create an environment with conda containing the NGS tools I want. I reinstalled conda to get a fresh start since I had many python packages in the base env. I also took the occasion to install the full anaconda3 instead of miniconda I had before now I have way enough space.
Although all the packages are available via bioconda the only 2 i can add to my env are fastqc and multiqc. Before that I could install sra-tools and fastqc in the base env with miniconda3.
Config: MacOS Monterey 12.1 M1 chip. migration from my old macbook air with the lastest time machine bkp. I uninstalled miniconda with the anaconda-clean procedure and after that I also removed a python 3.9.5 I had in the apps folder I had initially installed to start learning 1yr ago before knowing about conda.
Also to be mentioned in case it may help: anaconda-navigator was not installed by the Anaconda3-2022.05-MacOSX-arm64.pkg (sha256 integrity check was ok)
in following the check installation/navigator troubleshooting on anaconda website I came across an error upon launching spyder:
```(ModuleNotFoundError: No module named 'PyQt5.QtWebEngineWidgets')```
Does somebody know where this unability to find the packages comes?
Thank you for your help!
best, Daniel
Env creation command and console output:
(base) mymac:~ D________$ conda create --name ngstools fastqc multiqc sra-tools samtools bowtie2 hisat2 subread -c conda-forge -c bioconda -c bioconda/label/cf201901
Terminal output:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed
PackagesNotFoundError: The following packages are not available from current channels:
hisat2
subread
bowtie2
samtools
sra-tools
Current channels:
https://conda.anaconda.org/conda-forge/osx-arm64
https://conda.anaconda.org/conda-forge/noarch
https://conda.anaconda.org/bioconda/osx-arm64
https://conda.anaconda.org/bioconda/noarch
https://conda.anaconda.org/bioconda/label/cf201901/osx-arm64
https://conda.anaconda.org/bioconda/label/cf201901/noarch
https://repo.anaconda.com/pkgs/main/osx-arm64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/osx-arm64
https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Prefatory Comments
That is not a configuration I would recommend to bioinformatics users with M1 machines at the moment. There is no support yet for osx-arm64 in Bioconda. I expect one would find the least configuration issues by having everything in emulation (osx-64) from the base. Or you can continue with the osx-arm64 base (as you have now), but create new environments with the subdir setting. The answer below shows steps in this direction.
Also, I would very much avoid Anaconda/Miniconda. Mambaforge base is what I recommend to all bioinformaticians. And set the Bioconda channels globally.
Lastly, only advanced users should use channel labels (e.g., bioconda/label/cf201901). I realize they automatically show up on Anaconda Cloud, but they are only ever applicable in specialized circumstances, which most users will never encounter.
Immediate workaround
If you want to keep the osx-arm64 base, then you'll need to create a osx-64 environment for the NGS tools. I recommend the following protocol for best practice:
## create empty environment
conda create -n ngstools
## activate environment
conda activate ngstools
## configure architecture
conda config --env --set subdir osx-64
## configure channels for Bioconda
conda config --env --add channels defaults
conda config --env --add channels bioconda
conda config --env --add channels conda-forge
## install packages
conda install fastqc multiqc sra-tools samtools bowtie2 hisat2 subread
It's convoluted and, until osx-arm64 sees full support, you'd have to do this for every environment that requires emulation and Bioconda. Hence, why I recommend instead that you reinstall a osx-64 base (Mambaforge), which would just work.
Background
I have a multi-stage Dockerfile for a JupyterLab image. There are three stages:
server
kernel: mamba create -y -n /tmp/kernel-packages/myenv ...
runner:
FROM ... as runner
...
COPY --from=kernel /tmp/kernel-packages/ /opt/conda/envs
RUN conda config --append envs_dirs /opt/conda/envs/
...
Problem
In the resulting image, the preferred python -m pip works, but pip gives:
bash: /opt/conda/envs/myenv/bin/pip: /tmp/kernel-packages/myenv/bin/python3.9: bad interpreter: No such file or directory
The reason is that pip has #!/tmp/kernel-packages/myenv/bin/python3.9 as its shebang.
Expected
A behaviour like pip install --root /tmp/server-packages ..., which works perfectly fine in a COPY --from=server /tmp/server-packages /.
Additional information
Additionally, some other binaries, like curl or papermill also have wrong paths hardcoded by conda. I've read about Anaconda | Moving Conda Environments, but it seems like overkill to use conda-pack and conda-unpack.
Workaround
Simply create an env by name: mamba create -y -n myenv and then in the runner stage COPY --from=kernel /opt/conda/envs /opt/conda/envs.
Question
Is there a Conda (Mamba) equivalent to pip install's --root option? I'm not looking for a solution to the stated problem as I have I have already found a way that works for me. My interest is purely in the functionality of the conda binary.
The --prefix argument is the equivalent - just that some Conda packages use hardcoded paths, hence the issue you encounter.
conda-prefix-replacement
To properly move a Conda environment to a new prefix via a COPY operation one would need to run the conda-prefix-replacement tool (a.k.a., cpr) to ensure that all files with hardcoded paths get updated to the new location. Presumably, conda-pack is doing a similar thing, just under-the-hood.
For your purposes, you might consider pre-running cpr on the environment(s) in the kernel image so that they are ready to work in the deployed location. Though that would mean always COPYing to the same location.
See the cpr repository for details on use.
I'm getting a Malformed version string error with my conda. I have no idea how to debug this or how to check it.
Can anyone help? GitHub has talked about the issue but I haven't seen any fixes.
-bash-4.1$ conda install -c bioconda pysam
Solving environment: failed
1. CondaValueError: Malformed version string '~': invalid character(s).
In response to the comment below:
(mage_env) -bash-4.1$ echo $PATH
/usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/mage_env/bin:/usr/local/packages/jdk-8u121/bin/:/usr/local/bin:/usr/local/devel/ANNOTATION/rrichter/local/bin:/home/syooseph/utils/clustalw1.83:/usr/local/packages/gsl/bin:/usr/local/sge_current/bin/lx-amd64:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/devel/ANNOTATION/jespinoz/anaconda/bin:/usr/local/devel/ANNOTATION/jespinoz/Dropseq/:/usr/local/devel/ANNOTATION/jespinoz/Dropseq/Drop-seq_tools-1.13/
(mage_env) -bash-4.1$ conda info
active environment : mage_env
active env location : /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs/mage_env
shell level : 1
user config file : /home/jespinoz/.condarc
populated config files : /home/jespinoz/.condarc
conda version : 4.5.11
conda-build version : not installed
python version : 3.6.2.final.0
base environment : /usr/local/devel/ANNOTATION/jespinoz/anaconda (writable)
channel URLs : https://conda.anaconda.org/ursky/linux-64
https://conda.anaconda.org/ursky/noarch
https://conda.anaconda.org/bioconda/linux-64
https://conda.anaconda.org/bioconda/noarch
https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-64
https://repo.anaconda.com/pkgs/free/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
https://repo.anaconda.com/pkgs/pro/linux-64
https://repo.anaconda.com/pkgs/pro/noarch
package cache : /usr/local/devel/ANNOTATION/jespinoz/anaconda/pkgs
/home/jespinoz/.conda/pkgs
envs directories : /usr/local/devel/ANNOTATION/jespinoz/anaconda/envs
/home/jespinoz/.conda/envs
platform : linux-64
user-agent : conda/4.5.11 requests/2.14.2 CPython/3.6.2 Linux/2.6.32-696.18.7.el6.x86_64 centos/6.4 glibc/2.12
UID:GID : 3456:63
netrc file : None
offline mode : False
This looks like it was fixed with Conda 4.6.0. Upgrading your Conda should resolve the issue.
conda upgrade -n base conda
If you have trouble solving environment, I suggest trying
conda upgrade -n base -c defaults --override-channels conda
This will exclude all other channels during the solving process, and helps because having too many options to prune is one of the main reasons for slow solves.
I needed to remove conda-forge from my .condarc file. Then everything worked fine.
Had the same issue with conda 4.8.2 while I tried to create an environment from a yaml file.
loading a different, already existing environment and unloading it again did the trick in multiple occasions.
I got this error also when trying to install a package to a personal directory on an HPC, as in:
$ conda install -c dranew shapeit
Solving environment: failed
CondaValueError: Malformed version string '~': invalid character(s).
As a user of the HPC I'm not in control of the version of conda installed so had to find another solution.
This ended up working:
$ conda create --name shapeit -c dranew shapeit
or in generic terms,
$ conda create --name <packagename> -c <conda_channel_name> <packagename>
This tells conda to create an environment named packagename, then find the package in its channel (dranew for shapeit, bioconda for OP) and install it in that new environment.
I don't know if it makes a difference, but I ran this code when inside my .conda/envs directory.
Hope this helps someone else!
I had the similar problem
conda upgrade -n base conda
did pull the latest version but did not make the upgrade as I did not clear my previous install properly before installing latest anaconda.
Removing /root/anaconda3 ( default installation folder in my case ) &
removing /root/.conda & reinstalling the latest anaconda
Post that
conda upgrade -n base conda
would help you upgrade to 4.16+ version ( as of today, its latest )
One Additional Step which one must follow if they are importing env from one machine to this one:
remove specific version from conda package dependency, example here:
https://gitlab.com/mkuhring/TaxIt/-/commit/a8587e41bfff4c3362556c1c46d23d99ac3af069
I create a conda environment without specifying any packages using the following command:
conda create --name test_env
I can then use all the packages in the root environment inside test_env (but they do not appear in the outputs of conda list and conda env export). This is already unexpected to me but the real problems begin when I then install something inside that environment, e.g.:
conda install pywavelets
Afterwards, pywavelets is usable but all the other packages which are no dependencies of pywavelets disappear inside the environment (e.g. pandas). I don't understand why that happens. Does anybody have an explanation for that?
More importantly, what does this mean for best practices for working with conda environments? Should I always create my environments specifying at least python (conda create --name test_env python)? However, then I have to install everything by hand in that environment which is quite cumbersome. So, my idea now is to specify anaconda for all environments I create:
conda create --name test_env anaconda
The disadvantage, however, is that the list of dependencies listed by conda list and conda env export gets unnecessarily long (e.g. even listing the Anaconda Navigator). Does anybody have a better solution for this?
The reason you can use all the packages from the root environment when you don't specify a Python version during environment creation is because you're actually using the root environment's Python executable! You can check with which python or python -c "import sys; print(sys.executable)". See also my other answer here.
When you install pywavelets, one of the dependencies is (probably) Python, so a new Python executable is installed into your environment. Therefore, when you run Python, it only picks up the packages that are installed in the test_env.
If you want all of the packages from another environment, you can create a file that lists all the packages and then use that file to create a new environment, as detailed in the Conda docs: https://conda.io/docs/user-guide/tasks/manage-environments.html#building-identical-conda-environments
To summarize
conda list --explicit > spec-file.txt
conda create --name myenv --file spec-file.txt
or to install into an existing environment
conda install --name myenv --file spec-file.txt
Since that's just a text file, you can edit and remove any packages that you don't want.
How can I install a package under development to an Anaconda environment?
With pip:
pip install -e /path/to/mypackage
or with regular setuptools:
python /path/to/mypackage/setup.py develop
There is also conda develop available now.
http://conda.pydata.org/docs/commands/build/conda-develop.html
Update in 2019: conda develop hasn't been maintained and is not recommended. See https://github.com/conda/conda-build/issues/1992
Recommendation is to use python setup.py develop or pip install -e .
Using either of those will work with Anaconda. Make sure that you have pip or setuptools installed into the conda environment you want to install into, and that you have it activated.
This is the equivalent to pip install -e .
conda install conda-build
conda develop .
As explained in this gh issue thread, because of build isolation and dependency installation, Anaconda developers recommend using:
pip install --no-build-isolation --no-deps -e .
Build / Host Environment
To create build and host environments and a build script go to your recipe directory and use
conda debug /path/to/your/recipe-directory
as documented here. This will print an instructive message like
################################################################################
Build and/or host environments created for debugging. To enter a debugging environment:
cd /home/UserName/miniconda3/conda-bld/debug_1542385789430/work && source /home/UserName/miniconda3/conda-bld/debug_1542385789430/work/build_env_setup.sh
To run your build, you might want to start with running the conda_build.sh file.
################################################################################
(The message might tell you incorrectly, that it created a test environment.) Your source code has been copied to the .../work directory and there is also a conda_build.sh script. Note, that sourcing the build_env_setup.sh will load both build and host environments.
You can work on your code and your recipe and build with the conda_build.sh, but you won't get a proper conda package, as far as I know. When you are finished, you can remove the debug environment:
conda deactivate # maybe twice
conda build purge
Test Environment
To get the test environment, you have to build the package first and then debug that. This might be useful to fix your test files.
conda build /path/to/your/recipe-directory # creates mypackage*.tar.bz2
# find file location of mypackage*.tar.bz2 with:
conda search --info --use-local mypackage # look at the url row for the path
cd /path/to/miniconda3/conda-bld/linux-64/ # go to that path, can be different
conda debug mypackage*.tar.bz2
This will print e. g.:
################################################################################
Test environment created for debugging. To enter a debugging environment:
cd /home/UserName/miniconda3/conda-bld/debug_1542385789430/test_tmp && source /home/UserName/miniconda3/conda-bld/debug_1542385789430/work/conda_test_env_vars.sh
To run your tests, you might want to start with running the conda_test_runner.sh file.
################################################################################
Again, remove with
conda deactivate
conda build purge
Run Environment
This is actually no debugging, but the general process of building and installing a local package. With the run environment you can check, whether all dependencies are specified in the requirements/run section. Also pinning can be an issue.
(base) $ conda build /path/to/your/recipe-directory
(base) $ conda create --name package-env --use-local mypackage
(base) $ conda activate package-env
(package-env) $ python
>>> import mypackage
You can also list the dependencies of your package with (man page)
conda search --info --use-local mypackage
A last hint: If you want to know the versions of the dependencies and see, whether pinning works, try (man page)
conda render /path/to/your/recipe-directory