I'm reinstalling Conda after a PC factory reset and trying to re-create an old conda environment from a yml file that I created by
conda env export --prefix $path_to_old_env_dir > voice_dep.yml
The resulting yml file looks ok to me, here's what it looks like:
name: voiceeda
channels:
- defaults
- conda-forge
dependencies:
- ca-certificates=2022.12.7=h5b45459_0
- libsqlite=3.40.0=hcfcfb64_0
- openssl=1.1.1s=hcfcfb64_1
- pip=22.3.1=pyhd8ed1ab_0
- python=3.9.13=h6244533_2
- setuptools=66.1.1=pyhd8ed1ab_0
- sqlite=3.40.0=hcfcfb64_0
- tzdata=2022g=h191b570_0
- ucrt=10.0.22621.0=h57928b3_0
- vc=14.3=hb6edc58_10
- vs2015_runtime=14.34.31931=h4c5c07a_10
- wheel=0.38.4=pyhd8ed1ab_0
- pip:
- anyio==3.6.2
- argon2-cffi==21.3.0
- argon2-cffi-bindings==21.2.0
- arrow==1.2.3
...
but when I try to run
conda create -n voiceeda -f voice_dep.yml
The following odd error occurs.
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed
PackagesNotFoundError: The following packages are not available from current channels:
- voice_dep.yml
I'd understand if it wasn't finding a particular package, I can remove versions etc. if so, but why is it saying it can't find the yml file itself? I'm very confused, wondering if I missed a crucial setup step during conda installation or smth? I'm on Windows 10, and installed anaconda to a D drive (conda version 23.1.0 & Python 3.9.13.).
Any help would be much appreciated, thank you!
Related
When I try to run the experiment defined in this notebook in notebook, I encountered an error when it is creating the conda env. The error occurs when the below cell is executed:
from azureml.core import Experiment, ScriptRunConfig, Environment
from azureml.core.conda_dependencies import CondaDependencies
from azureml.widgets import RunDetails
# Create a Python environment for the experiment
sklearn_env = Environment("sklearn-env")
# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip'],
pip_packages=['azureml-defaults','azureml-dataprep[pandas]'])
sklearn_env.python.conda_dependencies = packages
# Get the training dataset
diabetes_ds = ws.datasets.get("diabetes dataset")
# Create a script config
script_config = ScriptRunConfig(source_directory=experiment_folder,
script='diabetes_training.py',
arguments = ['--regularization', 0.1, # Regularizaton rate parameter
'--input-data', diabetes_ds.as_named_input('training_data')], # Reference to dataset
environment=sklearn_env)
# submit the experiment
experiment_name = 'mslearn-train-diabetes'
experiment = Experiment(workspace=ws, name=experiment_name)
run = experiment.submit(config=script_config)
RunDetails(run).show()
run.wait_for_completion()
Everytime I run this, I always faced the issue of creating the conda env as below:
Creating conda environment...
Running: ['conda', 'env', 'create', '-p', '/home/azureuser/.azureml/envs/azureml_000000000000', '-f', 'azureml-environment-setup/mutated_conda_dependencies.yml']
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Installing pip dependencies: ...working...
Attempting to clean up partially built conda environment: /home/azureuser/.azureml/envs/azureml_000000000000
Remove all packages in environment /home/azureuser/.azureml/envs/azureml_000000000000:
Creating conda environment failed with exit code: -15
I could not find anything useful on the internet and this is not the only script where it fail. When I am try to run other experiments I have sometimes faced this issue. One solution which worked in the above case is I moved the pandas from pip to conda and it was able to create the coonda env. Example below:
# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip'],
pip_packages=['azureml-defaults','azureml-dataprep[pandas]'])
# Ensure the required packages are installed (we need scikit-learn, Azure ML defaults, and Azure ML dataprep)
packages = CondaDependencies.create(conda_packages=['scikit-learn','pip','pandas'],
pip_packages=['azureml-defaults','azureml-dataprep'])
The error message (or the logs from Azure) is also not much help. Would apprecite if a proper solution is available.
Edit: I have recently started learning to use Azure for Machine learning and so if I am not sure if I am missing something? I assume the example notebooks should work as is hence raised this question.
short answer
Totally been in your shoes before. This code sample seems a smidge out of date. Using this notebook as a reference, can you try the following?
packages = CondaDependencies.create(
pip_packages=['azureml-defaults','scikit-learn']
)
longer answer
Using pip with Conda is not always smooth sailing. In this instance, conda isn't reporting up the issue that pip is having. The solution is to create and test this environment locally where we can get more information, which will at least will give you a more informative error message.
Install anaconda or miniconda (or use an Azure ML Compute Instance which has conda pre-installed)
Make a file called environment.yml that looks like this
name: aml_env
dependencies:
- python=3.8
- pip=21.0.1
- pip:
- azureml-defaults
- azureml-dataprep[pandas]
- scikit-learn==0.24.1
Create this environment with the command conda env create -f environment.yml.
respond to any discovered error message
If there' no error, use this new environment.yml with Azure ML like so
sklearn_env = Environment.from_conda_specification(name = 'sklearn-env', file_path = './environment.yml')
more context
the error I'm guessing that's happening is when you reference a pip requirements file from a conda environment file. In this scenario, conda calls pip install -r requirements.txt and if that command errors out, conda can't report the error.
requirements.txt
scikit-learn==0.24.1
azureml-dataprep[pandas]
environment.yml
name: aml_env
dependencies:
- python=3.8
- pip=21.0.1
- pip:
- -rrequirements.txt
What worked for me looking at the previous notebook 05 - Train Models.ipynb:
packages = CondaDependencies.create(conda_packages=['pip', 'scikit-learn'],
pip_packages=['azureml-defaults'])
You have to:
Remove 'azureml-dataprep[pandas]' from pip_packages
Change the order of conda_packages - pip should go first
I tried to install graphviz but got a message that my environment is inconsistent. Looking at other posts I tried
conda install anaconda
conda install -c anaconda anconda
conda update --all
conda uninstall anaconda
but all failed.
When running "conda update --all", although it looks as if it fixed the problem, running the same command again gives the same output.
When running 'conda install anaconda' it ran for 6 hours without doing much. Eventually I stopped it. Here is the screen output:
(base) C:\Windows\system32>conda install anaconda
Collecting package metadata (current_repodata.json): done
Solving environment: |
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- defaults/win-64::ipykernel==5.3.4=py38h5ca1d4c_0
- conda-forge/noarch::ipympl==0.6.3=pyhd8ed1ab_0
- defaults/noarch::ipywidgets==7.6.3=pyhd3eb1b0_1
- conda-forge/noarch::nbclassic==0.2.5=pyhd8ed1ab_0
- defaults/noarch::nbclient==0.5.1=py_0
- defaults/win-64::nbconvert==6.0.7=py38_0
- defaults/win-64::notebook==6.1.4=py38_0
- defaults/win-64::widgetsnbextension==3.5.1=py38_0
- defaults/win-64::_ipyw_jlab_nb_ext_conf==0.1.0=py38_0
failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry
with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: \
The environment is inconsistent, please check the package plan carefully
The following packages are causing the inconsistency:
- defaults/win-64::ipykernel==5.3.4=py38h5ca1d4c_0
- conda-forge/noarch::ipympl==0.6.3=pyhd8ed1ab_0
- defaults/noarch::ipywidgets==7.6.3=pyhd3eb1b0_1
- conda-forge/noarch::nbclassic==0.2.5=pyhd8ed1ab_0
- defaults/noarch::nbclient==0.5.1=py_0
- defaults/win-64::nbconvert==6.0.7=py38_0
- defaults/win-64::notebook==6.1.4=py38_0
- defaults/win-64::widgetsnbextension==3.5.1=py38_0
- defaults/win-64::_ipyw_jlab_nb_ext_conf==0.1.0=py38_0
failed with initial frozen solve. Retrying with flexible solve.
Solving environment: /
Found conflicts! Looking for incompatible packages.
This can take several minutes. Press CTRL-C to abort.
Examining clyent: 23%|█████? | 41/176 [00:28<01:33, 1.44it/s]/
Examining libsodium: 38%|███████▌ | 67/176 [01:12<03:33, 1.96s/it]|
Examining conda-env: 41%|████████? | 72/176 [01:13<01:42, 1.02it/s]\
Examining m2w64-gmp: 56%|███████████? | 99/176 [03:40<03:11, 2.49s/it]\
Examining mkl-service: 65%|███████████ | 114/176 [04:38<06:29, 6.28s/it]\
Examining chardet: 69%|██████████████? | 121/176 [04:40<01:08, 1.24s/it]/
Examining conflict for six htmlmin testpath ipython console_shortcut wcwidth cf\
Examining conflict for six htmlmin testpath ipython console_shortcut wcwidth cf-
failed -
CondaError: KeyboardInterrupt
Terminate batch job (Y/N)? y
(base) C:\Windows\system32>
I an installing environment.yml file via
conda env create -f environment.yml
But I get
raise ReadTimeoutError(self._pool, None, 'Read timed out.')
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
failed
CondaEnvException: Pip failed
My environment.yml has a structure like this
name: relightable-nr
channels:
- pytorch
- defaults
dependencies:
- zlib=1.2.11=h7b6447c_3
- zstd=1.3.7=h0b5b093_0
- pip:
- absl-py==0.8.0
- astor==0.8.0
- astroid==2.3.3
- wrapt==1.11.2
- xarray==0.13.0
prefix: /root/anaconda3/envs/envn
I read How to solve ReadTimeoutError: HTTPSConnectionPool(host='pypi.python.org', port=443) with pip? and Pip Install Timeout Issue
I changed my conda default timeout to 300 but how to change pip timeout in my case here?
Pip will pull configuration options from a pip.conf/pip.inf (Unix/Win) file located in either a global, user, or environment scope, and settings such as timeout can be configured there. See the Pip User Guide section on Config Files.
While that answers the question proper, I would be remiss were I not to mention that all the packages listed in the YAML can come from Conda. A more appropriate solution would be to reconfigure the YAML to not hit PyPI in the first place, e.g.,
name: relightable-nr
channels:
- pytorch
- conda-forge
- defaults
dependencies:
- zlib=1.2.11=h7b6447c_3
- zstd=1.3.7=h0b5b093_0
- absl-py=0.8.0
- astor=0.8.0
- astroid=2.3.3
- wrapt=1.11.2
- xarray=0.13.0
but perhaps you abridged the YAML and left out packages that only have PyPI builds. Still, I would recommend getting everything possible from Conda.
You can use:
sudo pip install --default-timeout=100 <name_of_your_library>
I'm using two environment of conda. I can not intall packages in one env, while I can intall packages in the other environment.
The error massage is: 'solving environment: failed'
system: windows 10 x64
The error msg:
(py3env) C:\>conda install cython
Collecting package metadata (current_repodata.json): done
Solving environment: failed
Collecting package metadata (repodata.json): done
Solving environment: failed
PackagesNotFoundError: The following packages are not available from current channels:
- anaconda/pkgs/free/win-64::protobuf==3.2.0=py36_0 -> libprotobuf==3.2.0
- anaconda/pkgs/free/win-64::tensorflow==1.2.1=py36_0 -> backports.weakref==1.0rc1
- anaconda/pkgs/free/win-64::tensorflow==1.2.1=py36_0 -> bleach==1.5.0
- anaconda/pkgs/free/win-64::tensorflow==1.2.1=py36_0 -> html5lib==0.9999999
Current channels:
- https://repo.anaconda.com/pkgs/main/win-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/win-64
- https://repo.anaconda.com/pkgs/r/noarch
- https://repo.anaconda.com/pkgs/msys2/win-64
- https://repo.anaconda.com/pkgs/msys2/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
While the success info in another environment:
(py2env) C:\>conda install cython Collecting package metadata (current_repodata.json): done Solving environment: done
## Package Plan ##
environment location: C:\Users\sonic\Anaconda3\envs\py2env
added / updated specs:
- cython
The following packages will be downloaded:
package | build
---------------------------|-----------------
certifi-2019.6.16 | py27_0 151 KB
cython-0.29.11 | py27hc56fc5f_0 2.0 MB
------------------------------------------------------------
Total: 2.1 MB
The following NEW packages will be INSTALLED:
cython pkgs/main/win-64::cython-0.29.11-py27hc56fc5f_0
The following packages will be UPDATED:
certifi anaconda/pkgs/free::certifi-2016.2.28~ --> pkgs/main::certifi-2019.6.16-py27_0
Proceed ([y]/n)?
I think it is because you have packages installed from the "free" channel, but that channel has been removed. So conda is confused about what to do. You should read the blog post anaconda.com/why-we-removed-the-free-channel-in-conda-4-7 and temporarily add the "free" channel back to your configuration as described in that blog by running the command conda config --set restore_free_channel true. After you run that command, you can set the restore free channel back to false if you finished installing the Cython. Thanks for the comments of #darthbith
I'm struggling to figure out conda virtual environments on windows. All I want is to be able to have different versions of h2o installed at the same time because of their insane decision to not allow you to be able to load files saved in even the most minor different version.
I created a virtual environment by cloning my base anaconda:
conda create -n h203_14_0_7 --clone base
I then activated the virtual environment like so:
C:\ProgramData\Anaconda3\Scripts\activate h203_14_0_7
Now that I'm in the virtual environment (I see the (h203_14_0_7) at the beginning of the prompt), i want to uninstall the version of h2o in this virtual environment so I tried:
pip uninstall h2o
But this output
which to me looks like it's going to uninstall the global h2o rather than the virtual environment h2o. So I think it's using the global pip instead of the pip it should have cloned off the base. So how to I use the virtual environment pip to uninstall h2o just for my virtual environment and how can I be sure that it's doing the right thing?
I then ran
conda intall pip
and it seems that after that I was able to use pip to uninstall h2o only from the virtual environment (I hope). I then downloaded the older h2o version from here: https://github.com/h2oai/h2o-3/releases/tag/jenkins-rel-weierstrass-7
but when I try install it I get
(h203_14_0_7) C:\ProgramData\Anaconda3\envs\h203_14_0_7>pip install C:\Users\dan25\Downloads\h2o-3-jenkins-rel-weierstrass-7.tar.gz
Processing c:\users\dan25\downloads\h2o-3-jenkins-rel-weierstrass-7.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\ProgramData\Anaconda3\envs\h203_14_0_7\lib\tokenize.py", line 452, in open
buffer = _builtin_open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\dan25\\AppData\\Local\\Temp\\pip-sf7r_6pm-build\\setup.py'
So what now?
I had trouble (e.g. https://0xdata.atlassian.net/browse/PUBDEV-3370 ) getting that approach to ever work. It felt like some kind of global dependency was in there, somewhere.
So, I personally just uninstall, and install the desired version, as I need to move between versions. (Actually, I am more likely to use a different VirtualBox or AWS image for each.)
However I noticed searching for conda on the H2O jira that there is a lot of activity recently. They might all be pointing out the same bug you have found, but if so it sounds like it is something getting enough attention to get fixed.
Aside: finding old versions (and your edit showing install problems)
To find, e.g. 3.14.0.7, google it with "h2o". The top hit is http://h2o-release.s3.amazonaws.com/h2o/rel-weierstrass/7/index.html
The "rel-weierstrass" represents 3.14.0, and the 7 is in the URL. (I've yet to see a full list of all the rel-XXX names, but google will always find at least one in the series, even if it won't find the exact minor version.)
Download the zip file you find there. Inside you will find both an R package, and a whl package for Python. So unzip it, extract the one you want, then pip install it.
These zip files are always on S3 (AFAIK). The link you showed was a source snapshot, on github.
Install requirements:
pip install requests tabulate numpy scikit-learn
Extract the archive:
zcat h2o-3-jenkins-rel-weierstrass-7.tar.gz | tar xvf -
cd into Python directory and build:
cd h2o-py
../gradlew build
I have this working now. I think the trick is to make sure you do NOT have h2o installed on your base python. I did the following:
pip uninstall h2o
conda create --name h2o-base pip
conda activate h2o-base
conda install numpy
conda install pandas
conda install requests
conda install tabulate
conda install colorama
conda install future
conda install jupyter
python -m pip install ipykernel
conda deactivate
And now to install specific versions of h2o, you need to URL of the .whl file for that version and you can find a list of the URLs of all the old versions here: https://github.com/h2oai/h2o-3/blob/master/Changes.md
So for example to install version 3.18.0.8:
conda create --name h2o-3-18-0-8 --clone h2o-base
conda activate h2o-3-18-0-8
pip install http://h2o-release.s3.amazonaws.com/h2o/rel-wolpert/8/Python/h2o-3.18.0.8-py2.py3-none-any.whl
python -m ipykernel install --user --name h2o-3-18-0-8 --display-name "Python (h2o-3-18-0-8)"
or version 3.20.0.2 (make sure to conda deactivate first):
conda create --name h2o-3-20-0-2 --clone h2o-base
conda activate h2o-3-20-0-2
pip install http://h2o-release.s3.amazonaws.com/h2o/rel-wright/2/Python/h2o-3.20.0.2-py2.py3-none-any.whl
python -m ipykernel install --user --name h2o-3-20-0-2 --display-name "Python (h2o-3-20-0-2)"
This set-up allows me to have multiple versions of h2o installed on the same computer and if I have to use serialized models I just have to run python from the virtual environment with the correct version of h2o installed. I think this is preferable to uninstalling and reinstalling h2o each time.
Here is the environments.yml file if you want to skip all the manual installs above:
name: h2o-base
channels:
- conda-forge
- defaults
dependencies:
- asn1crypto=0.24.0=py37_1003
- backcall=0.1.0=py_0
- bleach=3.0.2=py_0
- ca-certificates=2018.10.15=ha4d7672_0
- certifi=2018.10.15=py37_1000
- cffi=1.11.5=py37hfa6e2cd_1001
- chardet=3.0.4=py37_1003
- colorama=0.4.0=py_0
- cryptography=2.3=py37h74b6da3_0
- cryptography-vectors=2.3.1=py37_1000
- decorator=4.3.0=py_0
- entrypoints=0.2.3=py37_1002
- future=0.16.0=py37_1002
- icu=58.2=vc14_0
- idna=2.7=py37_1002
- ipykernel=5.1.0=pyh24bf2e0_0
- ipython=7.0.1=py37h39e3cac_1000
- ipython_genutils=0.2.0=py_1
- ipywidgets=7.4.2=py_0
- jedi=0.13.1=py37_1000
- jinja2=2.10=py_1
- jpeg=9b=vc14_2
- jsonschema=2.6.0=py37_1002
- jupyter=1.0.0=py_1
- jupyter_client=5.2.3=py_1
- jupyter_console=6.0.0=py_0
- jupyter_core=4.4.0=py_0
- libflang=5.0.0=vc14_20180208
- libpng=1.6.34=vc14_0
- libsodium=1.0.16=vc14_0
- llvm-meta=5.0.0=0
- markupsafe=1.0=py37hfa6e2cd_1001
- mistune=0.8.4=py37hfa6e2cd_1000
- nbconvert=5.3.1=py_1
- nbformat=4.4.0=py_1
- notebook=5.7.0=py37_1000
- openblas=0.2.20=vc14_8
- openmp=5.0.0=vc14_1
- openssl=1.0.2p=hfa6e2cd_1001
- pandas=0.23.4=py37h830ac7b_1000
- pandoc=2.3.1=0
- pandocfilters=1.4.2=py_1
- parso=0.3.1=py_0
- pickleshare=0.7.5=py37_1000
- pip=18.1=py37_1000
- prometheus_client=0.4.2=py_0
- prompt_toolkit=2.0.6=py_0
- pycparser=2.19=py_0
- pygments=2.2.0=py_1
- pyopenssl=18.0.0=py37_1000
- pyqt=5.6.0=py37h764d66f_7
- pysocks=1.6.8=py37_1002
- python=3.7.0=hc182675_1005
- python-dateutil=2.7.3=py_0
- pytz=2018.5=py_0
- pywinpty=0.5.4=py37_1002
- pyzmq=17.1.2=py37hf576995_1001
- qt=5.6.2=vc14_1
- qtconsole=4.4.2=py_1
- requests=2.19.1=py37_1001
- send2trash=1.5.0=py_0
- setuptools=40.4.3=py37_0
- simplegeneric=0.8.1=py_1
- sip=4.18.1=py37h6538335_0
- six=1.11.0=py37_1001
- tabulate=0.8.2=py_0
- terminado=0.8.1=py37_1001
- testpath=0.4.2=py37_1000
- tornado=5.1.1=py37hfa6e2cd_1000
- traitlets=4.3.2=py37_1000
- urllib3=1.23=py37_1001
- vc=14=0
- vs2015_runtime=14.0.25420=0
- wcwidth=0.1.7=py_1
- webencodings=0.5.1=py_1
- wheel=0.32.1=py37_0
- widgetsnbextension=3.4.2=py37_1000
- win_inet_pton=1.0.1=py37_1002
- wincertstore=0.2=py37_1002
- winpty=0.4.3=4
- zeromq=4.2.5=vc14_2
- zlib=1.2.11=vc14_0
- blas=1.0=mkl
- icc_rt=2017.0.4=h97af966_0
- intel-openmp=2019.0=118
- m2w64-gcc-libgfortran=5.3.0=6
- m2w64-gcc-libs=5.3.0=7
- m2w64-gcc-libs-core=5.3.0=7
- m2w64-gmp=6.1.0=2
- m2w64-libwinpthread-git=5.0.0.4634.697f757=2
- mkl=2019.0=118
- mkl_fft=1.0.6=py37hdbbee80_0
- mkl_random=1.0.1=py37h77b88f5_1
- msys2-conda-epoch=20160418=1
- numpy=1.15.2=py37ha559c80_0
- numpy-base=1.15.2=py37h8128ebf_0