I have executed the pip install commands from RAPIDS into Google Colab ipynb jupyter notebook. They are:
pip install cudf-cu11 dask-cudf-cu11 --extra-index-url=https://pypi.ngc.nvidia.com
pip install cuml-cu11 --extra-index-url=https://pypi.ngc.nvidia.com
pip install cugraph-cu11 --extra-index-url=https://pypi.ngc.nvidia.com
Machine assigned by Colab:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version: 11.6 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 40C P0 26W / 70W | 0MiB / 15360MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
Error when execute every single (and separated) pip install:
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/, https://pypi.ngc.nvidia.com
Collecting cudf-cu11
Using cached cudf_cu11-23.2.0.tar.gz (6.5 kB)
error: subprocess-exited-with-error
× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Preparing metadata (setup.py) ... error
error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
Should obtain something like that but i didn't reach it
Try this:
!pip install cudf-cu11==22.12 rmm-cu11==22.12 --extra-index-url=https://pypi.ngc.nvidia.com/.
Source:
https://github.com/rapidsai/cudf/issues/12762#issuecomment-1427064693
Related
I'm unable to download packages from certain conda channels,but other channels work just fine.
To demonstrate, I made a new conda environment
conda create --name fastai2 python=3.7
Which runs just fine.
conda activate fastai2
Then, I want to install, NVIDIA rapids,
conda install -c rapidsai -c nvidia -c conda-forge -c defaults rapids=0.13 python=3.7
Which then proceeds to try to install lot of packages. However anything from the nvidia or rapidsai channels fail.
Downloading and Extracting Packages
nvstrings-0.13.0 | 129 KB | | 0%
dask-xgboost-0.2.0.d | 14 KB | | 0%
pygments-2.6.1 | 683 KB | ############################################################################# | 100%
libnvstrings-0.13.0 | 29.6 MB | | 0%
libnetcdf-4.7.4 | 1.3 MB | ############################################################################# | 100%
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.anaconda.org/rapidsai/linux-64/nvstrings-0.13.0-py37_0.tar.bz2>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
My condarc looks like this.
ssl_verify: /usr/local/share/ca-certificates/XXXXX_https_intercept.crt
allow_conda_downgrades: true
channels:
- conda-forge
- defaults
proxy_servers:
http: http://XXXXXXX
https: https://XXXXXXX
I can use wget or curl to down the packages, so no blocked websites via company firewall.
I also have problems with other channels. Like when I try to install fastai.
conda install -c pytorch -c fastai fastai
I've tried all sorts of stuff for the last 2 days. Like these issues which seem similar.
https://github.com/conda/conda/issues/8046
https://github.com/conda/conda/issues/6007
Any help would be appreciated.
Anthony
I had been using Anaconda with python 2.7
$ python
Python 2.7.14 |Anaconda custom (64-bit)| (default, Dec 7 2017, 17:05:42)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
When I decided to install tensorflow (since for some reason I had the non-gpu version)
The command I used was:
$ conda install -c anaconda tensorflow-gpu
However, after it was done (detail on output of this cmd to follow), I no longer had conda:
$ conda install -c conda-forge keras
Traceback (most recent call last):
File "/home/me/anaconda2/bin/conda", line 12, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
(Note: I also no longer had Keras) and was now running Python 3.7(!?):
$ python
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
What happened? How do I stop it from happening again? This happened once before and I ended up deleting all my anaconda files, then reinstalling. I don't want to make that a habit.
The output of my conda install was:
$ conda install -c anaconda tensorflow-gpu
Collecting package metadata: done
Solving environment: done
## Package Plan ##
environment location: /home/me/anaconda2
added / updated specs:
- tensorflow-gpu
The following packages will be downloaded:
package | build
---------------------------|-----------------
_tflow_190_select-0.0.1 | gpu 2 KB anaconda
absl-py-0.7.0 | py36_0 156 KB anaconda
astor-0.7.1 | py36_0 43 KB anaconda
c-ares-1.15.0 | h7b6447c_1 98 KB anaconda
ca-certificates-2018.12.5 | 0 123 KB anaconda
certifi-2018.11.29 | py36_0 146 KB anaconda
cudatoolkit-9.0 | h13b8566_0 340.4 MB anaconda
cudnn-7.1.2 | cuda9.0_0 367.8 MB anaconda
cupti-9.0.176 | 0 1.6 MB anaconda
curl-7.63.0 | hbc83047_1000 145 KB anaconda
gast-0.2.2 | py36_0 138 KB anaconda
git-2.11.1 | 0 9.5 MB anaconda
grpcio-1.16.1 | py36hf8bcb03_1 1.1 MB anaconda
krb5-1.16.1 | h173b8e3_7 1.4 MB anaconda
libcurl-7.63.0 | h20c2e04_1000 550 KB anaconda
libedit-3.1.20181209 | hc058e9b_0 188 KB anaconda
libssh2-1.8.0 | h1ba5d50_4 233 KB anaconda
markdown-3.0.1 | py36_0 107 KB anaconda
mkl_fft-1.0.10 | py36ha843d7b_0 170 KB anaconda
mkl_random-1.0.2 | py36hd81dba3_0 407 KB anaconda
ncurses-6.1 | he6710b0_1 958 KB anaconda
numpy-1.15.4 | py36h7e9f1db_0 47 KB anaconda
numpy-base-1.15.4 | py36hde5b4d6_0 4.3 MB anaconda
openssl-1.1.1 | h7b6447c_0 5.0 MB anaconda
pip-18.1 | py36_0 1.8 MB anaconda
protobuf-3.5.2 | py36hf484d3e_1 610 KB anaconda
python-3.6.8 | h0371630_0 34.4 MB anaconda
qt-4.8.7 | 2 34.1 MB anaconda
setuptools-40.6.3 | py36_0 625 KB anaconda
six-1.12.0 | py36_0 22 KB anaconda
sqlite-3.26.0 | h7b6447c_0 1.9 MB anaconda
tensorboard-1.9.0 | py36hf484d3e_0 3.3 MB anaconda
tensorflow-1.9.0 |gpu_py36h02c5d5e_1 3 KB anaconda
tensorflow-base-1.9.0 |gpu_py36h6ecc378_0 170.8 MB anaconda
tensorflow-gpu-1.9.0 | hf154084_0 2 KB anaconda
termcolor-1.1.0 | py36_1 7 KB anaconda
tk-8.6.8 | hbc83047_0 3.1 MB anaconda
werkzeug-0.14.1 | py36_0 423 KB anaconda
wheel-0.32.3 | py36_0 35 KB anaconda
------------------------------------------------------------
Total: 985.7 MB
The following NEW packages will be INSTALLED:
_tflow_190_select anaconda/linux-64::_tflow_190_select-0.0.1-gpu
c-ares anaconda/linux-64::c-ares-1.15.0-h7b6447c_1
cudatoolkit anaconda/linux-64::cudatoolkit-9.0-h13b8566_0
cudnn anaconda/linux-64::cudnn-7.1.2-cuda9.0_0
cupti anaconda/linux-64::cupti-9.0.176-0
krb5 anaconda/linux-64::krb5-1.16.1-h173b8e3_7
pip anaconda/linux-64::pip-18.1-py36_0
tensorflow-gpu anaconda/linux-64::tensorflow-gpu-1.9.0-hf154084_0
The following packages will be UPDATED:
absl-py conda-forge/noarch::absl-py-0.1.10-py~ --> anaconda/linux-64::absl-py-0.7.0-py36_0
ca-certificates conda-forge::ca-certificates-2018.11.~ --> anaconda::ca-certificates-2018.12.5-0
curl pkgs/main::curl-7.60.0-h84994c4_0 --> anaconda::curl-7.63.0-hbc83047_1000
gast 0.2.0-py27_0 --> 0.2.2-py36_0
grpcio pkgs/main::grpcio-1.12.1-py27hdbcaa40~ --> anaconda::grpcio-1.16.1-py36hf8bcb03_1
libcurl pkgs/main::libcurl-7.60.0-h1ad7b7a_0 --> anaconda::libcurl-7.63.0-h20c2e04_1000
libedit pkgs/main::libedit-3.1-heed3624_0 --> anaconda::libedit-3.1.20181209-hc058e9b_0
markdown conda-forge/noarch::markdown-2.6.11-p~ --> anaconda/linux-64::markdown-3.0.1-py36_0
mkl_fft pkgs/main::mkl_fft-1.0.6-py27hd81dba3~ --> anaconda::mkl_fft-1.0.10-py36ha843d7b_0
ncurses pkgs/main::ncurses-6.0-h9df7e31_2 --> anaconda::ncurses-6.1-he6710b0_1
openssl conda-forge::openssl-1.0.2p-h14c3975_~ --> anaconda::openssl-1.1.1-h7b6447c_0
protobuf conda-forge::protobuf-3.5.2-py27hd28b~ --> anaconda::protobuf-3.5.2-py36hf484d3e_1
python pkgs/main::python-2.7.14-h1571d57_29 --> anaconda::python-3.6.8-h0371630_0
setuptools pkgs/main::setuptools-38.4.0-py27_0 --> anaconda::setuptools-40.6.3-py36_0
six pkgs/main::six-1.11.0-py27h5f960f1_1 --> anaconda::six-1.12.0-py36_0
sqlite pkgs/main::sqlite-3.23.1-he433501_0 --> anaconda::sqlite-3.26.0-h7b6447c_0
tensorflow conda-forge::tensorflow-1.3.0-py27_0 --> anaconda::tensorflow-1.9.0-gpu_py36h02c5d5e_1
tk pkgs/main::tk-8.6.7-hc745277_3 --> anaconda::tk-8.6.8-hbc83047_0
wheel pkgs/main::wheel-0.30.0-py27h2bc6bb2_1 --> anaconda::wheel-0.32.3-py36_0
The following packages will be SUPERSEDED by a higher-priority channel:
certifi conda-forge::certifi-2018.11.29-py27_~ --> anaconda::certifi-2018.11.29-py36_0
git pkgs/main::git-2.17.0-pl526hb75a9fb_0 --> anaconda::git-2.11.1-0
libssh2 pkgs/main::libssh2-1.8.0-h9cfc8f7_4 --> anaconda::libssh2-1.8.0-h1ba5d50_4
mkl_random pkgs/main::mkl_random-1.0.2-py27hd81d~ --> anaconda::mkl_random-1.0.2-py36hd81dba3_0
numpy pkgs/main::numpy-1.15.4-py27h7e9f1db_0 --> anaconda::numpy-1.15.4-py36h7e9f1db_0
numpy-base pkgs/main::numpy-base-1.15.4-py27hde5~ --> anaconda::numpy-base-1.15.4-py36hde5b4d6_0
qt pkgs/main::qt-5.9.4-h4e5bff0_0 --> anaconda::qt-4.8.7-2
tensorflow-base pkgs/main::tensorflow-base-1.9.0-eige~ --> anaconda::tensorflow-base-1.9.0-gpu_py36h6ecc378_0
werkzeug pkgs/main::werkzeug-0.14.1-py27_0 --> anaconda::werkzeug-0.14.1-py36_0
The following packages will be DOWNGRADED:
astor 0.7.1-py27_0 --> 0.7.1-py36_0
tensorboard 1.10.0-py27hf484d3e_0 --> 1.9.0-py36hf484d3e_0
termcolor 1.1.0-py27_1 --> 1.1.0-py36_1
Proceed ([y]/n)? y
Downloading and Extracting Packages
tensorflow-gpu-1.9.0 | 2 KB | ########################################################################################################################################## | 100%
absl-py-0.7.0 | 156 KB | ########################################################################################################################################## | 100%
six-1.12.0 | 22 KB | ########################################################################################################################################## | 100%
git-2.11.1 | 9.5 MB | ########################################################################################################################################## | 100%
_tflow_190_select-0. | 2 KB | ########################################################################################################################################## | 100%
setuptools-40.6.3 | 625 KB | ########################################################################################################################################## | 100%
c-ares-1.15.0 | 98 KB | ########################################################################################################################################## | 100%
cupti-9.0.176 | 1.6 MB | ########################################################################################################################################## | 100%
libssh2-1.8.0 | 233 KB | ########################################################################################################################################## | 100%
gast-0.2.2 | 138 KB | ########################################################################################################################################## | 100%
ncurses-6.1 | 958 KB | ########################################################################################################################################## | 100%
protobuf-3.5.2 | 610 KB | ########################################################################################################################################## | 100%
tensorflow-base-1.9. | 170.8 MB | ########################################################################################################################################## | 100%
ca-certificates-2018 | 123 KB | ########################################################################################################################################## | 100%
python-3.6.8 | 34.4 MB | ########################################################################################################################################## | 100%
cudatoolkit-9.0 | 340.4 MB | ########################################################################################################################################## | 100%
qt-4.8.7 | 34.1 MB | ########################################################################################################################################## | 100%
sqlite-3.26.0 | 1.9 MB | ########################################################################################################################################## | 100%
astor-0.7.1 | 43 KB | ########################################################################################################################################## | 100%
tensorboard-1.9.0 | 3.3 MB | ########################################################################################################################################## | 100%
mkl_fft-1.0.10 | 170 KB | ########################################################################################################################################## | 100%
mkl_random-1.0.2 | 407 KB | ########################################################################################################################################## | 100%
certifi-2018.11.29 | 146 KB | ########################################################################################################################################## | 100%
wheel-0.32.3 | 35 KB | ########################################################################################################################################## | 100%
numpy-base-1.15.4 | 4.3 MB | ########################################################################################################################################## | 100%
numpy-1.15.4 | 47 KB | ########################################################################################################################################## | 100%
curl-7.63.0 | 145 KB | ########################################################################################################################################## | 100%
openssl-1.1.1 | 5.0 MB | ########################################################################################################################################## | 100%
tk-8.6.8 | 3.1 MB | ########################################################################################################################################## | 100%
libedit-3.1.20181209 | 188 KB | ########################################################################################################################################## | 100%
markdown-3.0.1 | 107 KB | ########################################################################################################################################## | 100%
werkzeug-0.14.1 | 423 KB | ########################################################################################################################################## | 100%
krb5-1.16.1 | 1.4 MB | ########################################################################################################################################## | 100%
termcolor-1.1.0 | 7 KB | ########################################################################################################################################## | 100%
pip-18.1 | 1.8 MB | ########################################################################################################################################## | 100%
libcurl-7.63.0 | 550 KB | ########################################################################################################################################## | 100%
tensorflow-1.9.0 | 3 KB | ########################################################################################################################################## | 100%
grpcio-1.16.1 | 1.1 MB | ########################################################################################################################################## | 100%
cudnn-7.1.2 | 367.8 MB | ########################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(OK - I see the change to Python 3.7 now, but that's still a nasty thing to have to be careful about. Is there some way to force it to leave my Python version alone?)
Cause
Changing Python versions without updating the conda package breaks Conda. . The Python version change (2.7.14 -> 3.6.8) created a situation where the new python has a new site-packages which no longer contains a conda package, whereas if you only update within 2.7.x, this wouldn't be an issue.
Conda includes both a set of binaries (e.g., what you're invoking when you type conda in a shell) and a Python package by the same name. The Python package is necessary for Conda as a whole to function and it get's loaded whenever you try to use conda.
It is problematic that many packages on Anaconda seem to be triggering Python version changes, but not subsequently triggering a conda package update. This sounds like something the dependency resolver is overlooking - i.e., default behavior should be to protect integrity of base environment where conda lives.
Trying to Recover
One possible route to recovery is to temporarily use micromamba (a standalone build of mamba) to repair the base environment. You can do all the following from any directory, so maybe use a temporary one or wherever you put downloads. Please report in the comments if this works or needs adjusting!
Installing Micromamba
Download the appropriate micromamba for your platform (here we'll use the latest linux-64 build). The actual binary will be at bin/micromamba:
# download and unpack
wget -qO- https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj bin/micromamba
Temporarily set MAMBA_ROOT_PREFIX to the base of your install. Typically this is the anaconda3 or miniconda3 folder; in this case, we'll use the path given by OP:
export MAMBA_ROOT_PREFIX=/home/me/anaconda2
Temporarily configured the shell to add the micromamba command:
eval "$(./bin/micromamba shell hook -s posix)"
Test that is works by checking the configuration information:
micromamba info
The key thing to check for is that base environment: correctly identifies to where your base env is and shows it as (writable). You should also see the pkgs folder in your base env in the package cache: .
Reinstall conda for the Current Python
(Re-)Install the conda package in the base env:
micromamba install -n base conda
Make sure that the build of Conda that is suggested corresponds to the version of Python currently installed. The --force-reinstall flag might be useful if it claims the requirement is already satisfied. Alternatively, try
micromamba upgrade -n base conda
Try a new shell and see if conda is working. You don't need to keep the micromamba around. However, I do enthusiastically encourage users to permanently install mamba (see next step).
(Optional) Install Mamba in base
Consider also installing Mamba directly in the base environment. It is a compiled (fast!) alternative frontend to Conda environment management.
micromamba install -n base mamba
One can then use mamba in most places where conda would be used.
Last Recourse
If all else fails you may just have to reinstall. Others have reported installing in other directories and being able to still use and access their environmentss.
Preventions
Avoiding Breakage through Better Practice
First, just a general (opinionated) recommendation: leverage virtual environments more. This isn't directly solving the problem, but it will help you have a workflow that is significantly less prone to encountering such pitfalls. You shouldn't have accepted such a huge change in the first place, not to base. Personally, I rarely install things in base outside of infrastructure (emacs, jupyter-related things, conda, etc.).1 Software packages go into project-specific or at least development-type environments.
For example, were I doing the install shown, I would have made a new environment for it
mamba create -n tf36 anaconda::tensorflow-gpu python=3.6
or whatever Python version you actually wish to work in.
Direct Solution: Pinning
Conda does support package pinning, and this is the more direct way to ensure you never ruin your base install again by transitioning Python 2 to 3. Namely, in the environment's conda-meta folder create a file, pinned and add the line
python 2.7.*
Note that some users have reported similar issues for 3.6 -> 3.7 transitions, so I believe including the minor version here is necessary. See the documentation on pinning.
[1] Note that I use a Miniforge variant (Mambaforge), not the Anaconda installer, so I have more control over base from the start.
I have solved this issue by removing any PYTHONHOME sys PATH(s).
I had been using Anaconda with python 2.7
$ python
Python 2.7.14 |Anaconda custom (64-bit)| (default, Dec 7 2017, 17:05:42)
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
When I decided to install tensorflow (since for some reason I had the non-gpu version)
The command I used was:
$ conda install -c anaconda tensorflow-gpu
However, after it was done (detail on output of this cmd to follow), I no longer had conda:
$ conda install -c conda-forge keras
Traceback (most recent call last):
File "/home/me/anaconda2/bin/conda", line 12, in <module>
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
(Note: I also no longer had Keras) and was now running Python 3.7(!?):
$ python
Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
What happened? How do I stop it from happening again? This happened once before and I ended up deleting all my anaconda files, then reinstalling. I don't want to make that a habit.
The output of my conda install was:
$ conda install -c anaconda tensorflow-gpu
Collecting package metadata: done
Solving environment: done
## Package Plan ##
environment location: /home/me/anaconda2
added / updated specs:
- tensorflow-gpu
The following packages will be downloaded:
package | build
---------------------------|-----------------
_tflow_190_select-0.0.1 | gpu 2 KB anaconda
absl-py-0.7.0 | py36_0 156 KB anaconda
astor-0.7.1 | py36_0 43 KB anaconda
c-ares-1.15.0 | h7b6447c_1 98 KB anaconda
ca-certificates-2018.12.5 | 0 123 KB anaconda
certifi-2018.11.29 | py36_0 146 KB anaconda
cudatoolkit-9.0 | h13b8566_0 340.4 MB anaconda
cudnn-7.1.2 | cuda9.0_0 367.8 MB anaconda
cupti-9.0.176 | 0 1.6 MB anaconda
curl-7.63.0 | hbc83047_1000 145 KB anaconda
gast-0.2.2 | py36_0 138 KB anaconda
git-2.11.1 | 0 9.5 MB anaconda
grpcio-1.16.1 | py36hf8bcb03_1 1.1 MB anaconda
krb5-1.16.1 | h173b8e3_7 1.4 MB anaconda
libcurl-7.63.0 | h20c2e04_1000 550 KB anaconda
libedit-3.1.20181209 | hc058e9b_0 188 KB anaconda
libssh2-1.8.0 | h1ba5d50_4 233 KB anaconda
markdown-3.0.1 | py36_0 107 KB anaconda
mkl_fft-1.0.10 | py36ha843d7b_0 170 KB anaconda
mkl_random-1.0.2 | py36hd81dba3_0 407 KB anaconda
ncurses-6.1 | he6710b0_1 958 KB anaconda
numpy-1.15.4 | py36h7e9f1db_0 47 KB anaconda
numpy-base-1.15.4 | py36hde5b4d6_0 4.3 MB anaconda
openssl-1.1.1 | h7b6447c_0 5.0 MB anaconda
pip-18.1 | py36_0 1.8 MB anaconda
protobuf-3.5.2 | py36hf484d3e_1 610 KB anaconda
python-3.6.8 | h0371630_0 34.4 MB anaconda
qt-4.8.7 | 2 34.1 MB anaconda
setuptools-40.6.3 | py36_0 625 KB anaconda
six-1.12.0 | py36_0 22 KB anaconda
sqlite-3.26.0 | h7b6447c_0 1.9 MB anaconda
tensorboard-1.9.0 | py36hf484d3e_0 3.3 MB anaconda
tensorflow-1.9.0 |gpu_py36h02c5d5e_1 3 KB anaconda
tensorflow-base-1.9.0 |gpu_py36h6ecc378_0 170.8 MB anaconda
tensorflow-gpu-1.9.0 | hf154084_0 2 KB anaconda
termcolor-1.1.0 | py36_1 7 KB anaconda
tk-8.6.8 | hbc83047_0 3.1 MB anaconda
werkzeug-0.14.1 | py36_0 423 KB anaconda
wheel-0.32.3 | py36_0 35 KB anaconda
------------------------------------------------------------
Total: 985.7 MB
The following NEW packages will be INSTALLED:
_tflow_190_select anaconda/linux-64::_tflow_190_select-0.0.1-gpu
c-ares anaconda/linux-64::c-ares-1.15.0-h7b6447c_1
cudatoolkit anaconda/linux-64::cudatoolkit-9.0-h13b8566_0
cudnn anaconda/linux-64::cudnn-7.1.2-cuda9.0_0
cupti anaconda/linux-64::cupti-9.0.176-0
krb5 anaconda/linux-64::krb5-1.16.1-h173b8e3_7
pip anaconda/linux-64::pip-18.1-py36_0
tensorflow-gpu anaconda/linux-64::tensorflow-gpu-1.9.0-hf154084_0
The following packages will be UPDATED:
absl-py conda-forge/noarch::absl-py-0.1.10-py~ --> anaconda/linux-64::absl-py-0.7.0-py36_0
ca-certificates conda-forge::ca-certificates-2018.11.~ --> anaconda::ca-certificates-2018.12.5-0
curl pkgs/main::curl-7.60.0-h84994c4_0 --> anaconda::curl-7.63.0-hbc83047_1000
gast 0.2.0-py27_0 --> 0.2.2-py36_0
grpcio pkgs/main::grpcio-1.12.1-py27hdbcaa40~ --> anaconda::grpcio-1.16.1-py36hf8bcb03_1
libcurl pkgs/main::libcurl-7.60.0-h1ad7b7a_0 --> anaconda::libcurl-7.63.0-h20c2e04_1000
libedit pkgs/main::libedit-3.1-heed3624_0 --> anaconda::libedit-3.1.20181209-hc058e9b_0
markdown conda-forge/noarch::markdown-2.6.11-p~ --> anaconda/linux-64::markdown-3.0.1-py36_0
mkl_fft pkgs/main::mkl_fft-1.0.6-py27hd81dba3~ --> anaconda::mkl_fft-1.0.10-py36ha843d7b_0
ncurses pkgs/main::ncurses-6.0-h9df7e31_2 --> anaconda::ncurses-6.1-he6710b0_1
openssl conda-forge::openssl-1.0.2p-h14c3975_~ --> anaconda::openssl-1.1.1-h7b6447c_0
protobuf conda-forge::protobuf-3.5.2-py27hd28b~ --> anaconda::protobuf-3.5.2-py36hf484d3e_1
python pkgs/main::python-2.7.14-h1571d57_29 --> anaconda::python-3.6.8-h0371630_0
setuptools pkgs/main::setuptools-38.4.0-py27_0 --> anaconda::setuptools-40.6.3-py36_0
six pkgs/main::six-1.11.0-py27h5f960f1_1 --> anaconda::six-1.12.0-py36_0
sqlite pkgs/main::sqlite-3.23.1-he433501_0 --> anaconda::sqlite-3.26.0-h7b6447c_0
tensorflow conda-forge::tensorflow-1.3.0-py27_0 --> anaconda::tensorflow-1.9.0-gpu_py36h02c5d5e_1
tk pkgs/main::tk-8.6.7-hc745277_3 --> anaconda::tk-8.6.8-hbc83047_0
wheel pkgs/main::wheel-0.30.0-py27h2bc6bb2_1 --> anaconda::wheel-0.32.3-py36_0
The following packages will be SUPERSEDED by a higher-priority channel:
certifi conda-forge::certifi-2018.11.29-py27_~ --> anaconda::certifi-2018.11.29-py36_0
git pkgs/main::git-2.17.0-pl526hb75a9fb_0 --> anaconda::git-2.11.1-0
libssh2 pkgs/main::libssh2-1.8.0-h9cfc8f7_4 --> anaconda::libssh2-1.8.0-h1ba5d50_4
mkl_random pkgs/main::mkl_random-1.0.2-py27hd81d~ --> anaconda::mkl_random-1.0.2-py36hd81dba3_0
numpy pkgs/main::numpy-1.15.4-py27h7e9f1db_0 --> anaconda::numpy-1.15.4-py36h7e9f1db_0
numpy-base pkgs/main::numpy-base-1.15.4-py27hde5~ --> anaconda::numpy-base-1.15.4-py36hde5b4d6_0
qt pkgs/main::qt-5.9.4-h4e5bff0_0 --> anaconda::qt-4.8.7-2
tensorflow-base pkgs/main::tensorflow-base-1.9.0-eige~ --> anaconda::tensorflow-base-1.9.0-gpu_py36h6ecc378_0
werkzeug pkgs/main::werkzeug-0.14.1-py27_0 --> anaconda::werkzeug-0.14.1-py36_0
The following packages will be DOWNGRADED:
astor 0.7.1-py27_0 --> 0.7.1-py36_0
tensorboard 1.10.0-py27hf484d3e_0 --> 1.9.0-py36hf484d3e_0
termcolor 1.1.0-py27_1 --> 1.1.0-py36_1
Proceed ([y]/n)? y
Downloading and Extracting Packages
tensorflow-gpu-1.9.0 | 2 KB | ########################################################################################################################################## | 100%
absl-py-0.7.0 | 156 KB | ########################################################################################################################################## | 100%
six-1.12.0 | 22 KB | ########################################################################################################################################## | 100%
git-2.11.1 | 9.5 MB | ########################################################################################################################################## | 100%
_tflow_190_select-0. | 2 KB | ########################################################################################################################################## | 100%
setuptools-40.6.3 | 625 KB | ########################################################################################################################################## | 100%
c-ares-1.15.0 | 98 KB | ########################################################################################################################################## | 100%
cupti-9.0.176 | 1.6 MB | ########################################################################################################################################## | 100%
libssh2-1.8.0 | 233 KB | ########################################################################################################################################## | 100%
gast-0.2.2 | 138 KB | ########################################################################################################################################## | 100%
ncurses-6.1 | 958 KB | ########################################################################################################################################## | 100%
protobuf-3.5.2 | 610 KB | ########################################################################################################################################## | 100%
tensorflow-base-1.9. | 170.8 MB | ########################################################################################################################################## | 100%
ca-certificates-2018 | 123 KB | ########################################################################################################################################## | 100%
python-3.6.8 | 34.4 MB | ########################################################################################################################################## | 100%
cudatoolkit-9.0 | 340.4 MB | ########################################################################################################################################## | 100%
qt-4.8.7 | 34.1 MB | ########################################################################################################################################## | 100%
sqlite-3.26.0 | 1.9 MB | ########################################################################################################################################## | 100%
astor-0.7.1 | 43 KB | ########################################################################################################################################## | 100%
tensorboard-1.9.0 | 3.3 MB | ########################################################################################################################################## | 100%
mkl_fft-1.0.10 | 170 KB | ########################################################################################################################################## | 100%
mkl_random-1.0.2 | 407 KB | ########################################################################################################################################## | 100%
certifi-2018.11.29 | 146 KB | ########################################################################################################################################## | 100%
wheel-0.32.3 | 35 KB | ########################################################################################################################################## | 100%
numpy-base-1.15.4 | 4.3 MB | ########################################################################################################################################## | 100%
numpy-1.15.4 | 47 KB | ########################################################################################################################################## | 100%
curl-7.63.0 | 145 KB | ########################################################################################################################################## | 100%
openssl-1.1.1 | 5.0 MB | ########################################################################################################################################## | 100%
tk-8.6.8 | 3.1 MB | ########################################################################################################################################## | 100%
libedit-3.1.20181209 | 188 KB | ########################################################################################################################################## | 100%
markdown-3.0.1 | 107 KB | ########################################################################################################################################## | 100%
werkzeug-0.14.1 | 423 KB | ########################################################################################################################################## | 100%
krb5-1.16.1 | 1.4 MB | ########################################################################################################################################## | 100%
termcolor-1.1.0 | 7 KB | ########################################################################################################################################## | 100%
pip-18.1 | 1.8 MB | ########################################################################################################################################## | 100%
libcurl-7.63.0 | 550 KB | ########################################################################################################################################## | 100%
tensorflow-1.9.0 | 3 KB | ########################################################################################################################################## | 100%
grpcio-1.16.1 | 1.1 MB | ########################################################################################################################################## | 100%
cudnn-7.1.2 | 367.8 MB | ########################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(OK - I see the change to Python 3.7 now, but that's still a nasty thing to have to be careful about. Is there some way to force it to leave my Python version alone?)
Cause
Changing Python versions without updating the conda package breaks Conda. . The Python version change (2.7.14 -> 3.6.8) created a situation where the new python has a new site-packages which no longer contains a conda package, whereas if you only update within 2.7.x, this wouldn't be an issue.
Conda includes both a set of binaries (e.g., what you're invoking when you type conda in a shell) and a Python package by the same name. The Python package is necessary for Conda as a whole to function and it get's loaded whenever you try to use conda.
It is problematic that many packages on Anaconda seem to be triggering Python version changes, but not subsequently triggering a conda package update. This sounds like something the dependency resolver is overlooking - i.e., default behavior should be to protect integrity of base environment where conda lives.
Trying to Recover
One possible route to recovery is to temporarily use micromamba (a standalone build of mamba) to repair the base environment. You can do all the following from any directory, so maybe use a temporary one or wherever you put downloads. Please report in the comments if this works or needs adjusting!
Installing Micromamba
Download the appropriate micromamba for your platform (here we'll use the latest linux-64 build). The actual binary will be at bin/micromamba:
# download and unpack
wget -qO- https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj bin/micromamba
Temporarily set MAMBA_ROOT_PREFIX to the base of your install. Typically this is the anaconda3 or miniconda3 folder; in this case, we'll use the path given by OP:
export MAMBA_ROOT_PREFIX=/home/me/anaconda2
Temporarily configured the shell to add the micromamba command:
eval "$(./bin/micromamba shell hook -s posix)"
Test that is works by checking the configuration information:
micromamba info
The key thing to check for is that base environment: correctly identifies to where your base env is and shows it as (writable). You should also see the pkgs folder in your base env in the package cache: .
Reinstall conda for the Current Python
(Re-)Install the conda package in the base env:
micromamba install -n base conda
Make sure that the build of Conda that is suggested corresponds to the version of Python currently installed. The --force-reinstall flag might be useful if it claims the requirement is already satisfied. Alternatively, try
micromamba upgrade -n base conda
Try a new shell and see if conda is working. You don't need to keep the micromamba around. However, I do enthusiastically encourage users to permanently install mamba (see next step).
(Optional) Install Mamba in base
Consider also installing Mamba directly in the base environment. It is a compiled (fast!) alternative frontend to Conda environment management.
micromamba install -n base mamba
One can then use mamba in most places where conda would be used.
Last Recourse
If all else fails you may just have to reinstall. Others have reported installing in other directories and being able to still use and access their environmentss.
Preventions
Avoiding Breakage through Better Practice
First, just a general (opinionated) recommendation: leverage virtual environments more. This isn't directly solving the problem, but it will help you have a workflow that is significantly less prone to encountering such pitfalls. You shouldn't have accepted such a huge change in the first place, not to base. Personally, I rarely install things in base outside of infrastructure (emacs, jupyter-related things, conda, etc.).1 Software packages go into project-specific or at least development-type environments.
For example, were I doing the install shown, I would have made a new environment for it
mamba create -n tf36 anaconda::tensorflow-gpu python=3.6
or whatever Python version you actually wish to work in.
Direct Solution: Pinning
Conda does support package pinning, and this is the more direct way to ensure you never ruin your base install again by transitioning Python 2 to 3. Namely, in the environment's conda-meta folder create a file, pinned and add the line
python 2.7.*
Note that some users have reported similar issues for 3.6 -> 3.7 transitions, so I believe including the minor version here is necessary. See the documentation on pinning.
[1] Note that I use a Miniforge variant (Mambaforge), not the Anaconda installer, so I have more control over base from the start.
I have solved this issue by removing any PYTHONHOME sys PATH(s).
I am having issues with conda. After running commands such as:
conda install -c /my_conda_channel numpy --offline --override-channels
the default conda channel has now become 'my_conda_channel' so that each subsequent package from this channel supercedes the default channel, which is not what I want. I did the former just for testing purposes.
How do I reset the channel behaviour?
Change the order from ~/.condarc so that defaults the first channel as
channels:
- defaults
- conda-forge
and add this line to it
channel_priority: true
or run the following code in command-line
conda config --set channel_priority true
then again run
conda update --all
Good Luck
Edited for new versions of conda. According to conda doc
As of version 4.6.0, Conda has a strict channel priority feature. Strict channel priority can dramatically speed up conda operations and also reduce package incompatibility problems. We recommend it as a default. However, it may break old environment files, so we plan to delay making it conda's out-of-the-box default until the next major version bump, conda 5.0.
channel_priority (ChannelPriority)
Accepts values of 'strict', 'flexible', and 'disabled'.
It still accepts the old values true and false
true := flexible
false := disabled
strict := this is a new value
Another option would be to move your channel to the bottom of the priority list.
Run the command....
conda config --append channels my_conda_channel
You should get a response like this...
Warning: 'my_conda_channel' already in 'channels' list, moving to the bottom
Verify...
conda config --get channels
Which should give you something like...
--add channels 'defaults' # highest priority
--add channels 'my_conda_channel' # lowest priority
Go to your home directory and open .condarc in an editor. Go to channels and edit the priority:
channels:
- defaults
- my_conda_channel
Now defaults will be preferred over my_conda_channel. You can also delete my_conda_channel.
Unfortunately none of the solutions worked for me as of April 2021.
There are several .condarc files that need to be edited, to ensure desired channel priority:
~/.condac. Here you'll find "global" channels that get prepended to all the other channels added manually (default channel e.g., or conda-forge got here somehow in my case, even though I did not add it manually). Changing/adding other channels via command line interface won't supercede the top priority of the channels listed here
.condarc files in the anaconda root dir. This is where channels added manually end up (I would also check ~/anaconda3/envs/{env_names}/ for environment specific .condarc files).
If you want full control over channel priority:
Clean channels sections in the ~/.condarc file (the top one).
Edit .condarc files per the desired channel priority manually, one file per environment (including base)
Set channel priority to true or strict
Check results of your edits with conda config --show channels
In the conda-meta directory for the enviornment, I added a file called 'pinned':
pinned:
tensorflow ==2.2.0
tensorflow-base ==2.2.0
tensorflow-datasets ==1.2.0
tensorflow-estimator ==2.2.0
Then, conda update --all didn't update the packages:
$ conda update --all
Collecting package metadata (current_repodata.json): - NVIDIA: no NVIDIA devices found
done
Solving environment: done
==> WARNING: A newer version of conda exists. <==
current version: 4.8.4
latest version: 4.9.1
Please update conda by running
$ conda update -n base -c defaults conda
## Package Plan ##
environment location: /home/ubuntu/anaconda2/envs/ai
The following packages will be downloaded:
package | build
---------------------------|-----------------
awscli-1.18.169 | py36h5fab9bb_0 1.8 MB conda-forge
boto3-1.16.9 | pyhd8ed1ab_0 70 KB conda-forge
botocore-1.19.9 | pyhd3deb0d_0 4.1 MB conda-forge
giflib-5.2.1 | h36c2ea0_2 77 KB conda-forge
hypothesis-5.41.0 | pyhd8ed1ab_0 222 KB conda-forge
jpeg-9d | h36c2ea0_0 264 KB conda-forge
libpng-1.6.37 | h21135ba_2 306 KB conda-forge
pandas-1.1.4 | py36hd87012b_0 10.5 MB conda-forge
tornado-6.1 | py36h1d69622_0 644 KB conda-forge
------------------------------------------------------------
Total: 17.9 MB
The following packages will be REMOVED:
keras-applications-1.0.8-py_1
The following packages will be UPDATED:
awscli 1.18.168-py36h5fab9bb_0 --> 1.18.169-py36h5fab9bb_0
boto3 1.16.8-pyhd8ed1ab_0 --> 1.16.9-pyhd8ed1ab_0
botocore 1.19.8-pyhd3deb0d_0 --> 1.19.9-pyhd3deb0d_0
hypothesis 5.40.0-pyhd8ed1ab_0 --> 5.41.0-pyhd8ed1ab_0
pandas 1.1.3-py36h66e3816_2 --> 1.1.4-py36hd87012b_0
tornado 6.0.4-py36h8c4c3a4_2 --> 6.1-py36h1d69622_0
The following packages will be DOWNGRADED:
giflib 5.2.1-h516909a_2 --> 5.2.1-h36c2ea0_2
jpeg 9d-h516909a_0 --> 9d-h36c2ea0_0
libpng 1.6.37-hed695b0_2 --> 1.6.37-h21135ba_2
Proceed ([y]/n)? y
Downloading and Extracting Packages
hypothesis-5.41.0 | 222 KB | ######################################### | 100%
tornado-6.1 | 644 KB | ######################################### | 100%
boto3-1.16.9 | 70 KB | ######################################### | 100%
libpng-1.6.37 | 306 KB | ######################################### | 100%
awscli-1.18.169 | 1.8 MB | ######################################### | 100%
jpeg-9d | 264 KB | ######################################### | 100%
botocore-1.19.9 | 4.1 MB | ######################################### | 100%
pandas-1.1.4 | 10.5 MB | ######################################### | 100%
giflib-5.2.1 | 77 KB | ######################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
$ conda list tensorflow
# packages in environment at /home/ubuntu/anaconda2/envs/ai:
#
# Name Version Build Channel
tensorflow 2.2.0 mkl_py36h5a57954_0
tensorflow-base 2.2.0 mkl_py36hd506778_0
tensorflow-datasets 1.2.0 py36_0 anaconda
tensorflow-estimator 2.2.0 pyh95af2a2_0 conda-forge
tensorflow-metadata 0.14.0 pyhe6710b0_1
You can change the channel priority as follows:
conda config --get
This will list all the channels from lowest to highest priority
add channels by using
conda config --add channels ---(your channel)
The last channel you add gets highest priority.. so maintain the order. you can add channels, even if you already have them so that the priority order gets changed
I am not able to install Py4j in Anaconda - Spyder.
I am working on Windows 32bit, Python 3.5 and anaconda 4.1.
I get the following error:
I tried multiple commands including
conda install py4j
I also run the following command:
C:\Users\360529>anaconda search -t conda py4j
Using Anaconda API: https://api.anaconda.org
Run 'anaconda show <USER/PACKAGE>' to get more details:
Packages:
Name | Version | Package Types | Platforms
------------------------- | ------ | --------------- | ---------------
Voskrese/py4j | 0.9 | conda | win-64
: Enables Python programs to dynamical
ly access arbitrary Java objects
anaconda-cluster/py4j | 0.9 | conda | linux-64, osx-64
: Enables Python programs to dynamical
ly access arbitrary Java objects
ashahba/py4j | 0.10.4 | conda | linux-64
auto/py4j | 0.8.1 | conda | linux-64, linux-32
: http://py4j.sourceforge.net/
blaze/py4j | 0.9 | conda | linux-64, osx-64
: Enables Python programs to dynamical
ly access arbitrary Java objects
chdoig/py4j | 0.8.1 | conda | osx-64
: Enables Python programs to dynamical
ly access arbitrary Java objects
conda-cluster/py4j | 0.8.2.1 | conda | linux-64, osx-64
: Enables Python programs to dynamical
ly access arbitrary Java objects
conda-forge/py4j | 0.10.4 | conda | linux-64, win-32,
win-64, osx-64
hargup/py4j | | conda | linux-64
: Enables Python programs to dynamical
ly access arbitrary Java objects
marciorf/py4j | 0.8.2.1 | conda | linux-64
: Enables Python programs to dynamical
ly access arbitrary Java objects
mutirri/py4j | 0.8.2.1 | conda | linux-64
quasiben/py4j | 0.10.1 | conda | linux-64, osx-64
sotera/py4j | 0.9 | conda | linux-64
: Enables Python programs to dynamical
ly access arbitrary Java objects
tapatk/py4j | 0.10.4 | conda | linux-64, win-64
Found 14 packages
It looks like Win-32bit version of 10.4 is available. But not sure why it is not installing.
You are typing:
conda install -c blaze py4j=0.10.4
This will try to fetch the package py4j v0.10.4 from the anaconda channel blaze. But as you can see from the output for anaconda search py4j:
conda-forge/py4j | 0.10.4 | conda | linux-64, win-32, win-64, osx-64
The Win-32 version is available on the conda channel conda-forge. So you need to type:
conda install -c conda-forge py4j=0.10.4