I've literally never seen anaconda's initial 'frozen solve' attempt work for installing a package. I was wondering is it possible to pass a CLI arg to disable this so that the installation process is faster?
An 'example' I'm sure we've all seen:
$ conda install any_package
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
No, there is currently no such CLI option.
However, if your concern is time spent solving, then you should be using Mamba.
Related
I'm trying to set-up by Apple Silicon Mac to be able to train tf models using its GPU.
I tried following the official instructions but I am getting the following error:
>>> conda install -c apple tensorflow-deps
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- tensorflow-deps
Current channels:
- https://conda.anaconda.org/apple/osx-64
- https://conda.anaconda.org/apple/noarch
- https://repo.anaconda.com/pkgs/main/osx-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/osx-64
- https://repo.anaconda.com/pkgs/r/noarch
- https://conda.anaconda.org/conda-forge/osx-64
- https://conda.anaconda.org/conda-forge/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Not sure if it should matter but I didn't install Miniforge because I already have Anaconda installed comprehensively. Surely, this can't be the reason? (I didn't want to install Miniforge not to mess up my env by having multiple Conda installations).
I did add the "apple" channel in the Navigator manually and the package does seem to be there:
https://anaconda.org/apple/tensorflow-deps
What am I missing here?
I also struggled with this for a while. The only way I was able to get a successful environment set up was indeed installing conda through mini forge. Based on this link I believe this is because of the other packages Anaconda pre-installs that are not ARM compatible.
I followed this thread to remove my Anaconda installation. Once that is done the instructions you linked should be successful.
I am attempting to install PyGMO on Mac OS X 11.2.2 (with Anaconda which I reinstalled so the Anaconda Navigator is now upgraded to 2.0.1.)
After the installation starts, it collects package metadata and reports it found package conflicts. How can I solve the conflict so that I can run PyGMO?
Here is the start:
$ conda install -c conda-forge pygmo
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: \
Found conflicts! Looking for incompatible packages.
After few hours, the Terminal returns a long report of conflicts and stops there. Here is a representative piece of output:
Package selectors2 conflicts for:
wurlitzer -> selectors2
spyder-kernels -> wurlitzer[version='>=1.0.3'] -> selectors2
Package mpmath conflicts for:
anaconda==2020.07=py38_0 -> sympy==1.6.1=py38_0 -> mpmath[version='>=0.19']
sympy -> mpmath[version='>=0.19']
anaconda==2020.07=py38_0 -> mpmath==1.1.0=py38_0
Package anyio conflicts for:
jupyterlab -> jupyter_server[version='>=1.4,<2'] -> anyio[version='>=2.0.2|>=2.0.2,<3']
jupyterlab_server -> jupyter_server[version='>=1.4,<2'] -> anyio[version='>=2.0.2|>=2.0.2,<3']
Package py-lief conflicts for:
conda-build -> py-lief
anaconda==2020.07=py38_0 -> py-lief==0.10.1=py38haf313ee_0
Note that strict channel priority may have removed packages required for satisfiability.
I followed the official installation guidelines and set the additional channel and its priority. I also checked this command but that is essentially the same thing. I also tried the installation commands from PyPI. And I tried this hint as well
There are two possible states:
Conda solver is correct. The previous package constraints you have in the environment are incompatible with installing pygmo. In that case, you either need to track down the conflicting constraints and try to manually loosen them (not recommended for Anaconda base), or you need to make a new environment:
conda create -n pygmo_env -c conda-forge pygmo
Include whatever other packages you need in there. E.g., ipykernel if you plan on using it as a Jupyter kernel.
Conda solver is bugging out. The solver is reporting trouble solving when it really shouldn't be. This happens, and especially happens when mixing channels (defaults and conda-forge). Many find Mamba, the drop-in replacement for Conda, to be more reliable (and definitely faster!).
conda install conda-forge::mamba
mamba install -c conda-forge pygmo
Unfortunately, it's hard to tell which state it's in. Many of us have been down the rabbit hole of trying to sort through the constraint reports and sometimes there really isn't a sensible conflict to be found. For practical purposes, I'd recommend trying out mamba. If it also fails, then at least you'll have good evidence that you're in state (1).
Additional Commentary
Despite upbeat documentation about installing from any channel in Anaconda Cloud, an Anaconda distribution is highly constrained - i.e., has too many packages - and only tests for co-installation of packages from the defaults channel. Additionally, Conda Forge and Anaconda have different build stacks, so there can be runtime package incompatibilities even when the solver allows co-installation.
Generally, I'd recommend making liberal use of environment creation. Aim to have separate environments for separate tasks/projects. If you plan on frequently using more than a vanilla Anaconda distribution, consider Miniforge or one of its variants. One can always create an Anaconda environment with conda create -n foo -c defaults anaconda.
When I try to run the experiment defined in this notebook in notebook, I encountered an error when it is creating the conda env. The error occurs when the below cell is executed:
from azureml.core import Experiment, ScriptRunConfig, Environment
from azureml.core.conda_dependencies import CondaDependencies
from azureml.widgets import RunDetails
# Create a Python environment for the experiment
sklearn_env = Environment("sklearn-env")
# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip'],
pip_packages=['azureml-defaults','azureml-dataprep[pandas]'])
sklearn_env.python.conda_dependencies = packages
# Get the training dataset
diabetes_ds = ws.datasets.get("diabetes dataset")
# Create a script config
script_config = ScriptRunConfig(source_directory=experiment_folder,
script='diabetes_training.py',
arguments = ['--regularization', 0.1, # Regularizaton rate parameter
'--input-data', diabetes_ds.as_named_input('training_data')], # Reference to dataset
environment=sklearn_env)
# submit the experiment
experiment_name = 'mslearn-train-diabetes'
experiment = Experiment(workspace=ws, name=experiment_name)
run = experiment.submit(config=script_config)
RunDetails(run).show()
run.wait_for_completion()
Everytime I run this, I always faced the issue of creating the conda env as below:
Creating conda environment...
Running: ['conda', 'env', 'create', '-p', '/home/azureuser/.azureml/envs/azureml_000000000000', '-f', 'azureml-environment-setup/mutated_conda_dependencies.yml']
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installing pip dependencies: ...working...
Attempting to clean up partially built conda environment: /home/azureuser/.azureml/envs/azureml_000000000000
Remove all packages in environment /home/azureuser/.azureml/envs/azureml_000000000000:
Creating conda environment failed with exit code: -15
I could not find anything useful on the internet and this is not the only script where it fail. When I am try to run other experiments I have sometimes faced this issue. One solution which worked in the above case is I moved the pandas from pip to conda and it was able to create the coonda env. Example below:
# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip'],
pip_packages=['azureml-defaults','azureml-dataprep[pandas]'])
# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip','pandas'],
pip_packages=['azureml-defaults','azureml-dataprep'])
The error message (or the logs from Azure) is also not much help. Would apprecite if a proper solution is available.
Edit: I have recently started learning to use Azure for Machine learning and so if I am not sure if I am missing something? I assume the example notebooks should work as is hence raised this question.
short answer
Totally been in your shoes before. This code sample seems a smidge out of date. Using this notebook as a reference, can you try the following?
packages = CondaDependencies.create(
pip_packages=['azureml-defaults','scikit-learn']
)
longer answer
Using pip with Conda is not always smooth sailing. In this instance, conda isn't reporting up the issue that pip is having. The solution is to create and test this environment locally where we can get more information, which will at least will give you a more informative error message.
Install anaconda or miniconda (or use an Azure ML Compute Instance which has conda pre-installed)
Make a file called environment.yml that looks like this
name: aml_env
dependencies:
- python=3.8
- pip=21.0.1
- pip:
- azureml-defaults
- azureml-dataprep[pandas]
- scikit-learn==0.24.1
Create this environment with the command conda env create -f environment.yml.
respond to any discovered error message
If there' no error, use this new environment.yml with Azure ML like so
sklearn_env = Environment.from_conda_specification(name = 'sklearn-env', file_path = './environment.yml')
more context
the error I'm guessing that's happening is when you reference a pip requirements file from a conda environment file. In this scenario, conda calls pip install -r requirements.txt and if that command errors out, conda can't report the error.
requirements.txt
scikit-learn==0.24.1
azureml-dataprep[pandas]
environment.yml
name: aml_env
dependencies:
- python=3.8
- pip=21.0.1
- pip:
- -rrequirements.txt
What worked for me looking at the previous notebook 05 - Train Models.ipynb:
packages = CondaDependencies.create(conda_packages=['pip', 'scikit-learn'],
pip_packages=['azureml-defaults'])
You have to:
Remove 'azureml-dataprep[pandas]' from pip_packages
Change the order of conda_packages - pip should go first
I have installed bioconda following the instructions at https://bioconda.github.io/user/install.html#set-up-channels. Then I tried
conda install bwa
conda install bcftools
conda install plink2
They all installed fine. However, when I tried
conda install bcftools-gtc2vcf-plugin
or
conda install -c bioconda bcftools-gtc2vcf-plugin
as instructed at https://bioconda.github.io/recipes/bcftools-gtc2vcf-plugin/README.html, I got errors as follows:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- bcftools-gtc2vcf-plugin
Current channels:
- https://conda.anaconda.org/bioconda/osx-64
- https://conda.anaconda.org/bioconda/noarch
- https://conda.anaconda.org/conda-forge/osx-64
- https://conda.anaconda.org/conda-forge/noarch
- https://repo.anaconda.com/pkgs/main/osx-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/osx-64
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
Any help would be highly appreciated.
Thanks in advance!
I would advise (as of 2020-01-06) not to use the bcftools-gtc2vcf-plugin as it is an old version missing many features compared to the current version. I would advise either to compile from source (https://github.com/freeseek/gtc2vcf) or alternatively to download pre-compiled binaries (https://personal.broadinstitute.org/giulio/gtc2vcf) that should work on systems with ≥GLIBC_2.3 installed (and making sure you are running the latest version of BCFtools)
If you get the error:
No functional bcftools plugins were found in
BCFTOOLS_PLUGINS="/Users/moxu/xbin/seq/bcftools/plugins".
- Is the plugin path correct?
- Run "bcftools plugin -lv" for more detailed error output.
Could not load "gtc2vcf".
(a bcftools plugin bug that the maintainers will fix soon), can you try to run one of the following commands instead:
$ bcftools plugin gtc2vcf -vv
$ bcftools +gtc2vcf -vv
$ bcftools plugin /Users/moxu/xbin/seq/bcftools/plugins/gtc2vcf.so -vv
$ bcftools +/Users/moxu/xbin/seq/bcftools/plugins/gtc2vcf.so -vv
You should get a reason for why the plugin is not loading. A typical error message could look like this:
/Users/moxu/xbin/seq/bcftools/plugins/gtc2vcf.so:
dlopen .. /lib64/libc.so.6: version `GLIBC_2.3' not found (required by /Users/moxu/xbin/seq/bcftools/plugins/gtc2vcf.so)
I cannot install mkl using conda on a IBM Power 9 machine. It seems that conda cannot find any suitable version for this machine since conda install mkl installs mkl on a RedHat machine without any error; though, on the IBM Power 9 with either conda install -c anaconda mkl or conda install mkl I get:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- mkl
Current channels:
- https://conda.anaconda.org/anaconda/linux-ppc64le
- https://conda.anaconda.org/anaconda/noarch
- https://repo.anaconda.com/pkgs/main/linux-ppc64le
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/linux-ppc64le
- https://repo.anaconda.com/pkgs/r/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
I also tried to download mkl from https://anaconda.org/anaconda/mkl/files and install it conda install on the machine, which was successfully installed. But, after that whenever I wanted to install a new package, I got an inconsistency error by conda:
Collecting package metadata (current_repodata.json): done
Solving environment: |
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- <unknown>/osx-64::mkl==2019.4=intel_233 failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: \
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- <unknown>/osx-64::mkl==2019.4=intel_233 failed with initial frozen solve. Retrying with flexible solve.
Solving environment: /
Found conflicts! Looking for incompatible packages. failed
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Package curl conflicts for:
cmake -> curl
anaconda==2019.07=py37_0 -> curl==7.65.2=hbc83047_0
pycurl -> curl=7.55
curl
Package libcurl conflicts for:
curl -> libcurl[version='7.59.0|7.60.0|7.61.0|7.61.1|7.62.0|7.63.0|7.63.0|7.64.0|7.64.1|7.65.2',build='h20c2e04_0|h20c2e04_0|h20c2e04_1000|h20c2e04_0|h20c2e04_0|h20c2e04_0|h1ad7b7a_0|h20c2e04_2']
cmake -> curl -> libcurl[version='7.59.0|7.60.0|7.61.0|7.61.1|7.62.0|7.63.0|7.63.0|7.64.0|7.64.1|7.65.2',build='h20c2e04_0|h20c2e04_0|h20c2e04_1000|h20c2e04_0|h20c2e04_0|h20c2e04_0|h1ad7b7a_0|h20c2e04_2']
anaconda==2019.07=py37_0 -> pycurl==7.43.0.3=py37h1ba5d50_0 -> libcurl[version='>=7.64.1,<8.0a0']
I appreciate any help or comment.