I'm using anaconda to regulate my environment,
for a project i have to use my GPU for network training.
I use pytorch for my project and i'm trying to get CUDA working.
I installed cudatoolkit, numba, cudnn
still, when i try this command:
torch.cuda.is_available()
I get "False" as output.
This is my environment:
# Name Version Build Channel
blas 1.0 mkl
bzip2 1.0.6 h470a237_2 conda-forge
ca-certificates 2018.03.07 0
cairo 1.14.12 he6fea26_5 conda-forge
certifi 2018.8.24 py35_1
cffi 1.11.5 py35he75722e_1
cloudpickle 0.5.5 py35_0
cudatoolkit 9.2 0 anaconda
cudnn 7.2.1 cuda9.2_0 anaconda
cycler 0.10.0 py_1 conda-forge
cython 0.28.5 py35hf484d3e_0 anaconda
dask-core 0.19.2 py35_0
dbus 1.13.0 h3a4f0e9_0 conda-forge
decorator 4.3.0 py35_0
expat 2.2.5 hfc679d8_2 conda-forge
ffmpeg 4.0.2 ha0c5888_1 conda-forge
fontconfig 2.13.1 h65d0f4c_0 conda-forge
freetype 2.9.1 h6debe1e_4 conda-forge
gettext 0.19.8.1 h5e8e0c9_1 conda-forge
giflib 5.1.4 h470a237_1 conda-forge
glib 2.55.0 h464dc38_2 conda-forge
gmp 6.1.2 hfc679d8_0 conda-forge
gnutls 3.5.19 h2a4e5f8_1 conda-forge
graphite2 1.3.12 hfc679d8_1 conda-forge
gst-plugins-base 1.12.5 hde13a9d_0 conda-forge
gstreamer 1.12.5 h61a6719_0 conda-forge
harfbuzz 1.9.0 h08d66d9_0 conda-forge
hdf5 1.10.2 hc401514_2 conda-forge
icu 58.2 hfc679d8_0 conda-forge
imageio 2.4.1 py35_0
intel-openmp 2019.0 118
jasper 1.900.1 hff1ad4c_5 conda-forge
jpeg 9c h470a237_1 conda-forge
kiwisolver 1.0.1 py35h2d50403_2 conda-forge
libedit 3.1.20170329 h6b74fdf_2
libffi 3.2.1 hd88cf55_4
libgcc-ng 8.2.0 hdf63c60_1
libgfortran 3.0.0 1 conda-forge
libgfortran-ng 7.3.0 hdf63c60_0
libiconv 1.15 h470a237_3 conda-forge
libopenblas 0.3.3 h5a2b251_3
libpng 1.6.35 ha92aebf_2 conda-forge
libstdcxx-ng 8.2.0 hdf63c60_1
libtiff 4.0.9 he6b73bb_2 conda-forge
libuuid 2.32.1 h470a237_2 conda-forge
libwebp 0.5.2 7 conda-forge
libxcb 1.13 h470a237_2 conda-forge
libxml2 2.9.8 h422b904_5 conda-forge
llvmlite 0.24.0 py35hdbcaa40_0
matplotlib 3.0.0 py35h0b34cb6_1 conda-forge
mkl 2019.0 118
mkl_fft 1.0.6 py35_0 conda-forge
mkl_random 1.0.1 py35_0 conda-forge
ncurses 6.1 hf484d3e_0
nettle 3.3 0 conda-forge
networkx 2.1 py35_0
ninja 1.8.2 py35h6bb024c_1
numba 0.39.0 py35h04863e7_0
numpy 1.15.2 py35h1d66e8a_0
numpy-base 1.15.2 py35h81de0dd_0
olefile 0.46 py35_0
openblas 0.2.20 8 conda-forge
opencv 3.4.1 py35h6fd60c2_1
opencv-python 3.4.3.18 <pip>
openh264 1.7.0 0 conda-forge
openssl 1.0.2p h14c3975_0
pandas 0.23.4 py35h04863e7_0
pcre 8.41 hfc679d8_3 conda-forge
pillow 5.2.0 py35heded4f4_0
Pillow 5.3.0 <pip>
pip 10.0.1 py35_0
pixman 0.34.0 h470a237_3 conda-forge
pthread-stubs 0.4 h470a237_1 conda-forge
pycparser 2.19 py35_0
pyparsing 2.2.2 py_0 conda-forge
pyqt 5.6.0 py35h8210e8a_7 conda-forge
python 3.5.6 hc3d631a_0
python-dateutil 2.7.3 py_0 conda-forge
pytorch 0.4.1 py35_py27__9.0.176_7.1.2_2 pytorch
pytz 2018.5 py35_0
pywavelets 1.0.0 py35hdd07704_0
qt 5.6.2 hf70d934_9 conda-forge
readline 7.0 h7b6447c_5
scikit-image 0.14.0 py35hf484d3e_1
scipy 1.1.0 py35hfa4b5c9_1
setuptools 40.2.0 py35_0
sip 4.18.1 py35hfc679d8_0 conda-forge
six 1.11.0 py35_1 conda-forge
sqlite 3.25.2 h7b6447c_0
tk 8.6.8 hbc83047_0
toolz 0.9.0 py35_0
torchvision 0.1.9 py35h72e4c6f_1 soumith
tornado 5.1.1 py35h470a237_0 conda-forge
wheel 0.31.1 py35_0
x264 1!152.20180717 h470a237_1 conda-forge
xorg-kbproto 1.0.7 h470a237_2 conda-forge
xorg-libice 1.0.9 h470a237_4 conda-forge
xorg-libsm 1.2.2 h8c8a85c_6 conda-forge
xorg-libx11 1.6.6 h470a237_0 conda-forge
xorg-libxau 1.0.8 h470a237_6 conda-forge
xorg-libxdmcp 1.1.2 h470a237_7 conda-forge
xorg-libxext 1.3.3 h470a237_4 conda-forge
xorg-libxrender 0.9.10 h470a237_2 conda-forge
xorg-renderproto 0.11.1 h470a237_2 conda-forge
xorg-xextproto 7.3.0 h470a237_2 conda-forge
xorg-xproto 7.0.31 h470a237_7 conda-forge
xz 5.2.4 h14c3975_4
zlib 1.2.11 ha838bed_2
My desktop has a NVIDIA GeForce GTX 970 (so it is cuda available)
Also for some reason, as you can see here:
My graphics card doesn't show, however when using the
lspci -v
command, i can see my graphics card there. Don't know if that has something to do with it. Anyone knows how i can fix this?
You need to install pytorch "in one go" using https://pytorch.org/get-started/locally/ to construct the anaconda command.
With the standard configuration in anaconda, you get:
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
(Please always check on https://pytorch.org/get-started/locally/ whether this command is still up to date.)
You seem to need the right cuda version 10.2 package to be aligned with what pytorch can handle. This is what is meant with #RussellGallop's helpful message.
We can see that installing pytorch and cuda separately is not recommended, and that Anaconda installation is recommended, against your answer:
Anaconda is our recommended package manager since it installs all dependencies.
Uninstall and install better than repair
In case of problems, you better uninstall all covered packages and apply https://pytorch.org/get-started/locally/ to get the command again, instead of trying to fix it with separate installations. Thus, if you want to uninstall, you need to use the exactly same command of the installation, but with "conda uninstall" instead.
For the example above, the uninstall command would be:
conda uninstall pytorch torchvision cudatoolkit=10.2 -c pytorch
This needed uninstall "in one go" again is another hint at the sensitive installation of pytorch, and that separate installation is risky. See How can l uninstall PyTorch?)
FIXED:
I installed CUDA seperately, not through anaconda and now it works. If anyone knows why it doesn't work when installing cuda in anaconda, feel free to answer
Your CUDA install needs to be compatible with your NVidia drivers (which aren't installed by Anaconda). I had a similar thing with Anaconda on Windows. I ignored the torch.cuda.is_available() check and tried to send a tensor to the GPU with:
device = torch.device("cuda")
torch.Tensor(1).to(device)
This gave a more helpful message:
The NVIDIA driver on your system is too old (found version 10010).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.
Related
This question already has answers here:
How to run pytorch with NVIDIA "cuda toolkit" version instead of the official conda "cudatoolkit" version?
(2 answers)
Closed 2 years ago.
Please feel free to vote "Reopen" at the bottom of this question. The reason is that I have marked this as a duplicate although the answers there are not clear enough for this question.
As soon as the question is reopened, I can add the the following answer: that cuDNN is an integrated part of the conda pytorch cudatoolkit installer and that it is listed together with pytorch, see py3.7_cuda102_cudnn7_0, and not as a stand-alone package that you might perhaps get by using pip or conda to install it explicitly (not recommended! See all answers of How to run pytorch with NVIDIA “cuda toolkit” version instead of the official conda “cudatoolkit” version?). With conda's cudatoolkit, cuDNN is automatically installed together with CUDA just for pytorch, and as a built-in package inside a conda package it does not show up.
I have installed pytorch using conda install pytorch torchvision cudatoolkit=10.2 -c pytorch.
https://github.com/pytorch/pytorch/issues/17445#issuecomment-466838819 says that cudNN is automatically included.
Trying to check the install according to How to get the CUDA version?, I used conda list cudnn to check, and it is not installed, even though cudNN is needed to use your gpu for ML, while conda list cuda works.
Here is the output of anaconda prompt of both commands.
(base) PS C:\Users\Admin> conda list cudnn
# packages in environment at C:\Users\Admin\anaconda3:
#
# Name Version Build Channel
(base) PS C:\Users\Admin> conda list cuda
# packages in environment at C:\Users\Admin\anaconda3:
#
# Name Version Build Channel
cudatoolkit 10.2.89 h74a9793_1
I find a 445 MB cudnn file "cudnn64_7.dll" in C:\Users\Admin\anaconda3\Lib\site-packages\torch\lib, and there are some small files in C:\Users\Admin\anaconda3\Lib\site-packages\torch\backends\cudnn and C:\Users\Admin\anaconda3\Lib\site-packages\torch\include\ATen.
Questions:
Is cudnn installed as a part of pytorch only?
Is there a way to get conda list cudnn to work?
####
Output of just conda list mentions cudnn at
pytorch 1.6.0 py3.7_cuda102_cudnn7_0 pytorch
Full list:
# Name Version Build Channel
_ipyw_jlab_nb_ext_conf 0.1.0 py37_0
alabaster 0.7.12 py37_0
anaconda 2020.07 py37_0
anaconda-client 1.7.2 py37_0
anaconda-navigator 1.9.12 py37_0
anaconda-project 0.8.4 py_0
argh 0.26.2 py37_0
asn1crypto 1.3.0 py37_1
astroid 2.4.2 py37_0
astropy 4.0.1.post1 py37he774522_1
atomicwrites 1.4.0 py_0
attrs 19.3.0 py_0
autopep8 1.5.3 py_0
babel 2.8.0 py_0
backcall 0.2.0 py_0
backports 1.0 py_2
backports.functools_lru_cache 1.6.1 py_0
backports.shutil_get_terminal_size 1.0.0 py37_2
backports.tempfile 1.0 py_1
backports.weakref 1.0.post1 py_1
bcrypt 3.1.7 py37he774522_1
beautifulsoup4 4.9.1 py37_0
bitarray 1.4.0 py37he774522_0
bkcharts 0.2 py37_0
blas 1.0 mkl
bleach 3.1.5 py_0
blosc 1.19.0 h7bd577a_0
bokeh 2.1.1 py37_0
boto 2.49.0 py37_0
bottleneck 1.3.2 py37h2a96729_1
brotlipy 0.7.0 py37he774522_1000
bzip2 1.0.8 he774522_0
ca-certificates 2020.6.24 0
certifi 2020.6.20 py37_0
cffi 1.14.0 py37h7a1dbc1_0
chardet 3.0.4 py37_1003
click 7.1.2 py_0
cloudpickle 1.5.0 py_0
clyent 1.2.2 py37_1
colorama 0.4.3 py_0
comtypes 1.1.7 py37_1001
conda 4.8.3 py37_0
conda-build 3.18.11 py37_0
conda-env 2.6.0 1
conda-package-handling 1.6.1 py37h62dcd97_0
conda-verify 3.4.2 py_1
console_shortcut 0.1.1 4
contextlib2 0.6.0.post1 py_0
cryptography 2.9.2 py37h7a1dbc1_0
cudatoolkit 10.2.89 h74a9793_1
curl 7.71.1 h2a8f88b_1
cycler 0.10.0 py37_0
cython 0.29.21 py37ha925a31_0
cytoolz 0.10.1 py37he774522_0
dask 2.20.0 py_0
dask-core 2.20.0 py_0
decorator 4.4.2 py_0
defusedxml 0.6.0 py_0
diff-match-patch 20200713 py_0
distributed 2.20.0 py37_0
docutils 0.16 py37_1
entrypoints 0.3 py37_0
et_xmlfile 1.0.1 py_1001
fastcache 1.1.0 py37he774522_0
filelock 3.0.12 py_0
flake8 3.8.3 py_0
flask 1.1.2 py_0
freetype 2.10.2 hd328e21_0
fsspec 0.7.4 py_0
future 0.18.2 py37_1
get_terminal_size 1.0.0 h38e98db_0
gevent 20.6.2 py37he774522_0
glob2 0.7 py_0
gmpy2 2.0.8 py37h0964b28_3
greenlet 0.4.16 py37he774522_0
h5py 2.10.0 py37h5e291fa_0
hdf5 1.10.4 h7ebc959_0
heapdict 1.0.1 py_0
html5lib 1.1 py_0
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.10 py_0
imageio 2.9.0 py_0
imagesize 1.2.0 py_0
importlib-metadata 1.7.0 py37_0
importlib_metadata 1.7.0 0
intel-openmp 2019.4 245
intervaltree 3.0.2 py_1
ipykernel 5.3.2 py37h5ca1d4c_0
ipython 7.16.1 py37h5ca1d4c_0
ipython_genutils 0.2.0 py37_0
ipywidgets 7.5.1 py_0
isort 4.3.21 py37_0
itsdangerous 1.1.0 py37_0
jdcal 1.4.1 py_0
jedi 0.17.1 py37_0
jinja2 2.11.2 py_0
joblib 0.16.0 py_0
jpeg 9b hb83a4c4_2
json5 0.9.5 py_0
jsonschema 3.2.0 py37_1
jupyter 1.0.0 py37_7
jupyter_client 6.1.6 py_0
jupyter_console 6.1.0 py_0
jupyter_core 4.6.3 py37_0
jupyterlab 2.1.5 py_0
jupyterlab_server 1.2.0 py_0
keyring 21.2.1 py37_0
kiwisolver 1.2.0 py37h74a9793_0
krb5 1.18.2 hc04afaa_0
lazy-object-proxy 1.4.3 py37he774522_0
libarchive 3.4.2 h5e25573_0
libcurl 7.71.1 h2a8f88b_1
libiconv 1.15 h1df5818_7
liblief 0.10.1 ha925a31_0
libllvm9 9.0.1 h21ff451_0
libpng 1.6.37 h2a8f88b_0
libsodium 1.0.18 h62dcd97_0
libspatialindex 1.9.3 h33f27b4_0
libssh2 1.9.0 h7a1dbc1_1
libtiff 4.1.0 h56a325e_1
libxml2 2.9.10 h464c3ec_1
libxslt 1.1.34 he774522_0
llvmlite 0.33.0 py37ha925a31_0
locket 0.2.0 py37_1
lxml 4.5.2 py37h1350720_0
lz4-c 1.9.2 h62dcd97_0
lzo 2.10 he774522_2
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs 5.3.0 7
m2w64-gcc-libs-core 5.3.0 7
m2w64-gmp 6.1.0 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
markupsafe 1.1.1 py37hfa6e2cd_1
matplotlib 3.2.2 0
matplotlib-base 3.2.2 py37h64f37c6_0
mccabe 0.6.1 py37_1
menuinst 1.4.16 py37he774522_1
mistune 0.8.4 py37hfa6e2cd_1001
mkl 2019.4 245
mkl-service 2.3.0 py37hb782905_0
mkl_fft 1.0.14 py37h6288b17_0
mkl_random 1.0.4 py37h343c172_0
mock 4.0.2 py_0
more-itertools 8.4.0 py_0
mpc 1.1.0 h7edee0f_1
mpfr 4.0.2 h62dcd97_1
mpir 3.0.0 hec2e145_1
mpmath 1.1.0 py37_0
msgpack-python 1.0.0 py37h74a9793_1
msys2-conda-epoch 20160418 1
multipledispatch 0.6.0 py37_0
navigator-updater 0.2.1 py37_0
nbconvert 5.6.1 py37_1
nbformat 5.0.7 py_0
networkx 2.4 py_1
ninja 1.9.0 py37h74a9793_0
nltk 3.5 py_0
nose 1.3.7 py37_1004
notebook 6.0.3 py37_0
numba 0.50.1 py37h47e9c7a_0
numexpr 2.7.1 py37h25d0782_0
numpy 1.17.0 py37h19fb1c0_0
numpy-base 1.17.0 py37hc3f5095_0
numpydoc 1.1.0 py_0
olefile 0.46 py37_0
openpyxl 3.0.4 py_0
openssl 1.1.1g he774522_0
packaging 20.4 py_0
pandas 1.0.5 py37h47e9c7a_0
pandoc 2.10 0
pandocfilters 1.4.2 py37_1
paramiko 2.7.1 py_0
parso 0.7.0 py_0
partd 1.1.0 py_0
path 13.1.0 py37_0
path.py 12.4.0 0
pathlib2 2.3.5 py37_1
pathtools 0.1.2 py_1
patsy 0.5.1 py37_0
pep8 1.7.1 py37_0
pexpect 4.8.0 py37_1
pickleshare 0.7.5 py37_1001
pillow 7.2.0 py37hcc1f983_0
pip 20.1.1 py37_1
pkginfo 1.5.0.1 py37_0
pluggy 0.13.1 py37_0
ply 3.11 py37_0
powershell_shortcut 0.0.1 3
prometheus_client 0.8.0 py_0
prompt-toolkit 3.0.5 py_0
prompt_toolkit 3.0.5 0
psutil 5.7.0 py37he774522_0
py 1.9.0 py_0
py-lief 0.10.1 py37ha925a31_0
pybson 0.5.9 pypi_0 pypi
pyclustering 0.9.3.1 pypi_0 pypi
pycodestyle 2.6.0 py_0
pycosat 0.6.3 py37he774522_0
pycparser 2.20 py_2
pycrypto 2.6.1 py37he774522_10
pycurl 7.43.0.5 py37h7a1dbc1_0
pydocstyle 5.0.2 py_0
pyflakes 2.2.0 py_0
pygments 2.6.1 py_0
pylint 2.5.3 py37_0
pymongo 3.9.0 py37ha925a31_0
pynacl 1.4.0 py37h62dcd97_1
pyodbc 4.0.30 py37ha925a31_0
pyopenssl 19.1.0 py_1
pyparsing 2.4.7 py_0
pyqt 5.9.2 py37h6538335_2
pyreadline 2.1 py37_1
pyrsistent 0.16.0 py37he774522_0
pysocks 1.7.1 py37_1
pytables 3.6.1 py37h1da0976_0
pytest 5.4.3 py37_0
python 3.7.7 h81c818b_4
python-dateutil 2.8.1 py_0
python-jsonrpc-server 0.3.4 py_1
python-language-server 0.34.1 py37_0
python-libarchive-c 2.9 py_0
pytorch 1.6.0 py3.7_cuda102_cudnn7_0 pytorch
pytz 2020.1 py_0
pywavelets 1.1.1 py37he774522_0
pywin32 227 py37he774522_1
pywin32-ctypes 0.2.0 py37_1001
pywinpty 0.5.7 py37_0
pyyaml 5.3.1 py37he774522_1
pyzmq 19.0.1 py37ha925a31_1
qdarkstyle 2.8.1 py_0
qt 5.9.7 vc14h73c81de_0
qtawesome 0.7.2 py_0
qtconsole 4.7.5 py_0
qtpy 1.9.0 py_0
regex 2020.6.8 py37he774522_0
requests 2.24.0 py_0
rope 0.17.0 py_0
rtree 0.9.4 py37h21ff451_1
ruamel_yaml 0.15.87 py37he774522_1
scikit-image 0.16.2 py37h47e9c7a_0
scikit-learn 0.23.1 py37h25d0782_0
scipy 1.5.0 py37h9439919_0
seaborn 0.10.1 py_0
send2trash 1.5.0 py37_0
setuptools 49.2.0 py37_0
simplegeneric 0.8.1 py37_2
singledispatch 3.4.0.3 py37_0
sip 4.19.8 py37h6538335_0
six 1.15.0 py_0
snappy 1.1.8 h33f27b4_0
snowballstemmer 2.0.0 py_0
sortedcollections 1.2.1 py_0
sortedcontainers 2.2.2 py_0
soupsieve 2.0.1 py_0
sphinx 3.1.2 py_0
sphinxcontrib 1.0 py37_1
sphinxcontrib-applehelp 1.0.2 py_0
sphinxcontrib-devhelp 1.0.2 py_0
sphinxcontrib-htmlhelp 1.0.3 py_0
sphinxcontrib-jsmath 1.0.1 py_0
sphinxcontrib-qthelp 1.0.3 py_0
sphinxcontrib-serializinghtml 1.1.4 py_0
sphinxcontrib-websupport 1.2.3 py_0
spyder 4.1.4 py37_0
spyder-kernels 1.9.2 py37_0
sqlalchemy 1.3.18 py37he774522_0
sqlite 3.32.3 h2a8f88b_0
statsmodels 0.11.1 py37he774522_0
sympy 1.6.1 py37_0
tbb 2020.0 h74a9793_0
tblib 1.6.0 py_0
terminado 0.8.3 py37_0
testpath 0.4.4 py_0
threadpoolctl 2.1.0 pyh5ca1d4c_0
tk 8.6.10 he774522_0
toml 0.10.1 py_0
toolz 0.10.0 py_0
torchvision 0.7.0 py37_cu102 pytorch
tornado 6.0.4 py37he774522_1
tqdm 4.47.0 py_0
traitlets 4.3.3 py37_0
typed-ast 1.4.1 py37he774522_0
typing_extensions 3.7.4.2 py_0
ujson 1.35 py37hfa6e2cd_0
unicodecsv 0.14.1 py37_0
urllib3 1.25.9 py_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.16.27012 hf0eaf9b_3
watchdog 0.10.3 py37_0
wcwidth 0.2.5 py_0
webencodings 0.5.1 py37_1
werkzeug 1.0.1 py_0
wheel 0.34.2 py37_0
widgetsnbextension 3.5.1 py37_0
win_inet_pton 1.1.0 py37_0
win_unicode_console 0.5 py37_0
wincertstore 0.2 py37_0
winpty 0.4.3 4
wrapt 1.11.2 py37he774522_0
xlrd 1.2.0 py37_0
xlsxwriter 1.2.9 py_0
xlwings 0.19.5 py37_0
xlwt 1.3.0 py37_0
xmltodict 0.12.0 py_0
xz 5.2.5 h62dcd97_0
yaml 0.2.5 he774522_0
yapf 0.30.0 py_0
zeromq 4.3.2 ha925a31_2
zict 2.0.0 py_0
zipp 3.1.0 py_0
zlib 1.2.11 h62dcd97_4
zope 1.0 py37_1
zope.event 4.4 py37_0
zope.interface 4.7.1 py37he774522_0
zstd 1.4.5 ha9fde0e_0
You can try installing CuDNN explicitly by using conda install -c anaconda cudnn. Then conda list cudnn should show CuDNN.
I was trying to run dask-on-yarn with my research group's Hadoop cluster.
I tried each of the following instructions:
dd.read_parquet('hdfs://file.parquet', engine='fastparquet')
dd.read_parquet('hdfs://file.parquet', engine='pyarrow')
dd.read_csv('hdfs://file.csv')
Each time, the following error message occurs:
~/miniconda3/envs/dask/lib/python3.8/site-packages/fsspec/core.py in get_fs_token_paths(urlpath, mode, num, name_function, storage_options, protocol)
521 path = cls._strip_protocol(urlpath)
522 update_storage_options(options, storage_options)
--> 523 fs = cls(**options)
524
525 if "w" in mode:
~/miniconda3/envs/dask/lib/python3.8/site-packages/fsspec/spec.py in __call__(cls, *args, **kwargs)
52 return cls._cache[token]
53 else:
---> 54 obj = super().__call__(*args, **kwargs)
55 # Setting _fs_token here causes some static linters to complain.
56 obj._fs_token_ = token
~/miniconda3/envs/dask/lib/python3.8/site-packages/fsspec/implementations/hdfs.py in __init__(self, host, port, user, kerb_ticket, driver, extra_conf, **kwargs)
42 AbstractFileSystem.__init__(self, **kwargs)
43 self.pars = (host, port, user, kerb_ticket, driver, extra_conf)
---> 44 self.pahdfs = HadoopFileSystem(
45 host=host,
46 port=port,
~/miniconda3/envs/dask/lib/python3.8/site-packages/pyarrow/hdfs.py in __init__(self, host, port, user, kerb_ticket, driver, extra_conf)
38 _maybe_set_hadoop_classpath()
39
---> 40 self._connect(host, port, user, kerb_ticket, extra_conf)
41
42 def __reduce__(self):
~/miniconda3/envs/dask/lib/python3.8/site-packages/pyarrow/io-hdfs.pxi in pyarrow.lib.HadoopFileSystem._connect()
~/miniconda3/envs/dask/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
OSError: Getting symbol hdfsNewBuilderfailed
How should I resolve this problem?
My Environment
Here are my packages in this conda env:
# Name Version Build Channel
_libgcc_mutex 0.1 main
abseil-cpp 20200225.2 he1b5a44_0 conda-forge
arrow-cpp 0.17.1 py38h1234567_9_cpu conda-forge
attrs 19.3.0 py_0
aws-sdk-cpp 1.7.164 hc831370_1 conda-forge
backcall 0.2.0 py_0
blas 1.0 mkl
bleach 3.1.5 py_0
bokeh 2.1.1 py38_0
boost-cpp 1.72.0 h7b93d67_1 conda-forge
brotli 1.0.7 he6710b0_0
brotlipy 0.7.0 py38h7b6447c_1000
bzip2 1.0.8 h7b6447c_0
c-ares 1.15.0 h7b6447c_1001
ca-certificates 2020.6.24 0
certifi 2020.6.20 py38_0
cffi 1.14.0 py38he30daa8_1
chardet 3.0.4 py38_1003
click 7.1.2 py_0
cloudpickle 1.4.1 py_0
conda-pack 0.4.0 py_0
cryptography 2.9.2 py38h1ba5d50_0
curl 7.71.0 hbc83047_0
cytoolz 0.10.1 py38h7b6447c_0
dask 2.19.0 py_0
dask-core 2.19.0 py_0
dask-yarn 0.8.1 py38h32f6830_0 conda-forge
decorator 4.4.2 py_0
defusedxml 0.6.0 py_0
distributed 2.19.0 py38_0
entrypoints 0.3 py38_0
fastparquet 0.3.2 py38heb32a55_0
freetype 2.10.2 h5ab3b9f_0
fsspec 0.7.4 py_0
gflags 2.2.2 he6710b0_0
glog 0.4.0 he6710b0_0
grpc-cpp 1.30.0 h9ea6770_0 conda-forge
grpcio 1.27.2 py38hf8bcb03_0
heapdict 1.0.1 py_0
icu 67.1 he1b5a44_0 conda-forge
idna 2.10 py_0
importlib-metadata 1.7.0 py38_0
importlib_metadata 1.7.0 0
intel-openmp 2020.1 217
ipykernel 5.3.0 py38h5ca1d4c_0
ipython 7.16.1 py38h5ca1d4c_0
ipython_genutils 0.2.0 py38_0
jedi 0.17.1 py38_0
jinja2 2.11.2 py_0
jpeg 9b h024ee3a_2
json5 0.9.5 py_0
jsonschema 3.2.0 py38_0
jupyter_client 6.1.3 py_0
jupyter_core 4.6.3 py38_0
jupyterlab 2.1.5 py_0
jupyterlab_server 1.1.5 py_0
krb5 1.18.2 h173b8e3_0
ld_impl_linux-64 2.33.1 h53a641e_7
libcurl 7.71.0 h20c2e04_0
libedit 3.1.20191231 h7b6447c_0
libevent 2.1.10 hcdb4288_1 conda-forge
libffi 3.3 he6710b0_1
libgcc-ng 9.1.0 hdf63c60_0
libgfortran-ng 7.3.0 hdf63c60_0
libllvm9 9.0.1 h4a3c616_0
libpng 1.6.37 hbc83047_0
libprotobuf 3.12.3 hd408876_0
libsodium 1.0.18 h7b6447c_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 9.1.0 hdf63c60_0
libtiff 4.1.0 h2733197_1
llvmlite 0.33.0 py38hd408876_0
locket 0.2.0 py38_1
lz4-c 1.9.2 he6710b0_0
markupsafe 1.1.1 py38h7b6447c_0
mistune 0.8.4 py38h7b6447c_1000
mkl 2020.1 217
mkl-service 2.3.0 py38he904b0f_0
mkl_fft 1.1.0 py38h23d657b_0
mkl_random 1.1.1 py38h0573a6f_0
msgpack-python 1.0.0 py38hfd86e86_1
nbconvert 5.6.1 py38_0
nbformat 5.0.7 py_0
ncurses 6.2 he6710b0_1
notebook 6.0.3 py38_0
numba 0.50.1 py38h0573a6f_0
numpy 1.18.5 py38ha1c710e_0
numpy-base 1.18.5 py38hde5b4d6_0
olefile 0.46 py_0
openssl 1.1.1g h7b6447c_0
packaging 20.4 py_0
pandas 1.0.5 py38h0573a6f_0
pandoc 2.9.2.1 0
pandocfilters 1.4.2 py38_1
parquet-cpp 1.5.1 2 conda-forge
parso 0.7.0 py_0
partd 1.1.0 py_0
pexpect 4.8.0 py38_0
pickleshare 0.7.5 py38_1000
pillow 7.1.2 py38hb39fc2d_0
pip 20.1.1 py38_1
prometheus_client 0.8.0 py_0
prompt-toolkit 3.0.5 py_0
protobuf 3.12.3 py38he6710b0_0
psutil 5.7.0 py38h7b6447c_0
ptyprocess 0.6.0 py38_0
pyarrow 0.17.1 py38h1234567_9_cpu conda-forge
pycparser 2.20 py_0
pygments 2.6.1 py_0
pyopenssl 19.1.0 py38_0
pyparsing 2.4.7 py_0
pyrsistent 0.16.0 py38h7b6447c_0
pysocks 1.7.1 py38_0
python 3.8.3 hcff3b4d_2
python-dateutil 2.8.1 py_0
python_abi 3.8 1_cp38 conda-forge
pytz 2020.1 py_0
pyyaml 5.3.1 py38h7b6447c_1
pyzmq 19.0.1 py38he6710b0_1
re2 2020.07.01 he1b5a44_0 conda-forge
readline 8.0 h7b6447c_0
requests 2.24.0 py_0
send2trash 1.5.0 py38_0
setuptools 47.3.1 py38_0
six 1.15.0 py_0
skein 0.8.0 py38h32f6830_1 conda-forge
snappy 1.1.8 he6710b0_0
sortedcontainers 2.2.2 py_0
sqlite 3.32.3 h62c20be_0
tbb 2020.0 hfd86e86_0
tblib 1.6.0 py_0
terminado 0.8.3 py38_0
testpath 0.4.4 py_0
thrift 0.13.0 py38he6710b0_0
thrift-cpp 0.13.0 h62aa4f2_2 conda-forge
tk 8.6.10 hbc83047_0
toolz 0.10.0 py_0
tornado 6.0.4 py38h7b6447c_1
traitlets 4.3.3 py38_0
typing_extensions 3.7.4.2 py_0
urllib3 1.25.9 py_0
wcwidth 0.2.5 py_0
webencodings 0.5.1 py38_1
wheel 0.34.2 py38_0
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
zeromq 4.3.2 he6710b0_2
zict 2.0.0 py_0
zipp 3.1.0 py_0
zlib 1.2.11 h7b6447c_3
zstd 1.4.4 h0b5b093_3
The Hadoop cluster is running version Hadoop 2.7.0-mapr-1607.
The Cluster object is created with:
# Create a cluster where each worker has two cores and eight GiB of memory
cluster = YarnCluster(
environment='conda-env-packed-for-worker-nodes.tar.gz',
worker_env={
# See https://github.com/dask/dask-yarn/pull/30#issuecomment-434001858
'ARROW_LIBHDFS_DIR': '/opt/mapr/hadoop/hadoop-0.20.2/c++/Linux-amd64-64/lib',
},
)
Suspected Cause
I suspect that the version mismatch between the hadoop-0.20.2 in the ARROW_LIBHDFS_DIR environmental variable and the hadoop CLI version Hadoop 2.7.0 might be a cause of the problem.
I had to manually specify pyarrow to use this file (using this setup: https://stackoverflow.com/a/62749053/1147061). The required file, libhdfs.so, is not provided under /opt/mapr/hadoop/hadoop-2.7.0/. Installing libhdfs3 via conda install -c conda-forge libhdfs3 does not resolve the requirement, either.
Might this be the problem?
(a part answer)
To use libhdfs3 (which is poorly maintained these days), you would need to call
dd.read_csv('hdfs://file.csv', storage_options={'driver': 'libhdfs3'})
and, of course, install libhdfs3. This will not help with the hadoop library option, they are independent code paths.
I also suspect that getting the JNI libhdfs (without the "3") working is a case of locating the right .so file.
I have created an environment but when I try to export it with
conda env export --name ENVNAME > ENVNAME.yml
I get the following error message:
InvalidVersionSpec: Invalid version '(>=': unable to convert to expression tree: ['(']
In response to the question from FlyingTeller, here is what conda list gives (sorry for the long list):
# Name Version Build Channel
altair 3.2.0 py38_0
appdirs 1.4.3 pypi_0 pypi
appnope 0.1.0 py38_0
asn1crypto 1.3.0 py38_0
attrs 19.3.0 py_0
backcall 0.1.0 py38_0
black 19.10b0 pypi_0 pypi
blas 1.0 mkl
bleach 3.1.0 py_0
ca-certificates 2020.1.1 0
certifi 2019.11.28 py38_0
cffi 1.14.0 py38hb5b8e2f_0
chardet 3.0.4 py38_1003
click 7.0 pypi_0 pypi
cloudpickle 1.3.0 py_0
cryptography 2.8 py38ha12b0ac_0
cycler 0.10.0 py38_0
decorator 4.4.1 py_0
defusedxml 0.6.0 py_0
entrypoints 0.3 py38_0
et_xmlfile 1.0.1 py38_0
ezc3d 1.2.4 py38hbf1eeb5_0 conda-forge
freetype 2.9.1 hb4e5f40_0
idna 2.8 py38_1000
importlib_metadata 1.5.0 py38_0
intel-openmp 2020.0 166
ipykernel 5.1.4 py38h39e3cac_0
ipython 7.12.0 py38h5ca1d4c_0
ipython-genutils 0.2.0 pypi_0 pypi
ipython_genutils 0.2.0 py38_0
jdcal 1.4.1 py_0
jedi 0.16.0 py38_0
jinja2 2.11.1 py_0
joblib 0.14.1 py_0
jsonschema 3.2.0 py38_0
jupyter_client 5.3.4 py38_0
jupyter_core 4.6.1 py38_0
kiwisolver 1.1.0 py38ha1b3eb9_0 conda-forge
libcxx 9.0.1 1 conda-forge
libedit 3.1.20181209 hb402a30_0
libffi 3.2.1 h475c297_4
libgfortran 3.0.1 h93005f0_2
libpng 1.6.37 ha441bb4_0
libsodium 1.0.16 h3efe00b_0
littleutils 0.2.2 py_0 conda-forge
llvm-openmp 4.0.1 hcfea43d_1
markupsafe 1.1.1 py38h1de35cc_0
matplotlib 3.1.3 py38_0
matplotlib-base 3.1.3 py38h9aa3819_0
mistune 0.8.4 py38h1de35cc_1000
mkl 2019.4 233
mkl-service 2.3.0 py38hfbe908c_0
mkl_fft 1.0.15 py38h5e564d8_0
mkl_random 1.1.0 py38h6440ff4_0
more-itertools 8.2.0 py_0
mpmath 1.1.0 py38_0
nb-black 1.0.7 pypi_0 pypi
nb_conda_kernels 2.2.2 py38_0 conda-forge
nbconvert 5.6.1 py38_0
nbformat 5.0.4 py_0
ncurses 6.1 h0a44026_1
notebook 6.0.3 py38_0
numpy 1.18.1 py38h7241aed_0
numpy-base 1.18.1 py38h6575580_1
openpyxl 3.0.3 py_0
openssl 1.1.1d h1de35cc_4
outdated 0.2.0 py_0 conda-forge
pandas 1.0.1 py38h6c726b0_0
pandas-flavor 0.2.0 py_0 conda-forge
pandoc 2.2.3.2 0
pandocfilters 1.4.2 py38_1
parso 0.6.1 py_0
pathspec 0.7.0 pypi_0 pypi
patsy 0.5.1 py38_0
peakutils 1.3.2 py_0 conda-forge
pexpect 4.8.0 py38_0
pickleshare 0.7.5 py38_1000
pingouin 0.3.2 py_0 conda-forge
pip 20.0.2 py38_1
prometheus_client 0.7.1 py_0
prompt_toolkit 3.0.3 py_0
ptyprocess 0.6.0 py38_0
pycparser 2.19 py_0
pygments 2.5.2 py_0
pyopenssl 19.1.0 py38_0
pyparsing 2.4.6 py_0
pyrsistent 0.15.7 py38h1de35cc_0
pysocks 1.7.1 py38_0
python 3.8.1 h359304d_1
python-dateutil 2.8.1 py_0
pytz 2019.3 py_0
pyzmq 18.1.1 py38h0a44026_0
readline 7.0 h1de35cc_5
regex 2020.1.8 pypi_0 pypi
requests 2.22.0 py38_1
scikit-learn 0.22.1 py38h27c97d8_0
scipy 1.4.1 py38h44e99c9_0
seaborn 0.10.0 py_0
send2trash 1.5.0 py38_0
setuptools 45.2.0 py38_0
six 1.14.0 py38_0
spyder-kernels 1.8.1 py38_0
sqlite 3.31.1 ha441bb4_0
statsmodels 0.11.0 py38h1de35cc_0
terminado 0.8.3 py38_0
testpath 0.4.4 py_0
tk 8.6.8 ha441bb4_0
toml 0.10.0 pypi_0 pypi
toolz 0.10.0 py_0
tornado 6.0.3 py38h1de35cc_3
traitlets 4.3.3 py38_0
typed-ast 1.4.1 pypi_0 pypi
urllib3 1.25.8 py38_0
wcwidth 0.1.8 py_0
webencodings 0.5.1 py38_1
wheel 0.34.2 py38_0
wurlitzer 2.0.0 py38_0
xarray 0.15.0 py_0
xlrd 1.2.0 py_0
xlwt 1.3.0 py38_0
xz 5.2.4 h1de35cc_4
zeromq 4.3.1 h0a44026_3
zipp 2.2.0 py_0
zlib 1.2.11 h1de35cc_3
I had run into a similar issue: it is due to some (everlasting) incompatibilities between conda and pip. In my case the pip package nb-black was causing the error: running pip uninstall nb-black followed by conda env export > environment.yml solved it: uninstalling black before exporting your environment might help.
I had a similar issue, which was caused by my own package, created with setuptools and distributed as wheel.
I have used a wrong format, with extra trailing dot, to specify required dependency versions in setup.py of my package.
# setup.py
import setuptools
setuptools.setup(
name="mypackage",
version="1.0.0",
python_requires=">=3.7",
install_requires=[
"pandas>=0.25",
"pyodbc>=4.", # Warning! This should be without trailing dot "pyodbc>=4",
"requests>=2.26"
]
)
In the package installed in a conda environment this information gets saved to %CONDA_PREFIX%/Lib/site-packages/mypackage-1.0.0.dist-info/METADATA. This is where command conda env export fails at reading package versions.
On windows you can use this method, suggested by RLashofRegas at
https://github.com/conda/conda/issues/9624#issuecomment-801623523
In Windows PowerShell you input
cd C:\Users\<user>\Anaconda3\envs\<env name>\Lib\site-packages
then
Get-ChildItem -File -Recurse -Filter METADATA | Select-String "4.7.0<4.8.0" | Select-Object -Unique Path
where you put your error inside the quotation marks, e.g. 4.7.0<4.8.0
This gives the path to the package file, that causes all the troubles.
You can either change it's METADATA, or uninstall.
I had a similar issue. I used the export command in debug mode to identify the package causing the problem. Then, uninstalling the package fixed the issue.
conda env export -vv
I installed PySide2 from conda-forge into an virtual environment named "Qt4Python".
It works if I directly type in the Anaconda prompt (in python):
import PySide2
from PySide2.QtWidgets import QApplication, QLabel
However, if I write these 2 lines in a script file named test1.py, then also from Anaconda prompt call:
python test1.py
Then it produces:
AttributeError: module 'PySide2' has no attribute 'QtWidgets'
What I was missing here ? Below is the package list of the environment.
(Qt4Python) C:\Users\XuanPhu\Desktop\ST_project>conda list
# packages in environment at C:\Users\XuanPhu\AppData\Local\conda\conda\envs\Qt4Python:
#
# Name Version Build Channel
backcall 0.1.0 py37_0
ca-certificates 2018.11.29 ha4d7672_0 conda-forge
certifi 2018.11.29 py37_1000 conda-forge
colorama 0.4.1 py37_0
decorator 4.3.0 py37_0
icu 58.2 ha66f8fd_1
ipython 7.2.0 py37h39e3cac_0
ipython_genutils 0.2.0 py37_0
jedi 0.13.2 py37_0
jpeg 9c hfa6e2cd_1001 conda-forge
libiconv 1.15 hfa6e2cd_1004 conda-forge
libpng 1.6.36 h7602738_1000 conda-forge
libxml2 2.9.8 h9ce36c8_1005 conda-forge
libxslt 1.1.32 heafd4d3_1002 conda-forge
openssl 1.0.2p hfa6e2cd_1002 conda-forge
parso 0.3.1 py37_0
pickleshare 0.7.5 py37_0
pip 18.1 py37_1000 conda-forge
prompt_toolkit 2.0.7 py37_0
pygments 2.3.1 py37_0
pyside2 5.6.0a1 py37h20f30f1_1003 conda-forge
python 3.7.1 hc182675_1000 conda-forge
qt 5.6.2 h2639256_8 conda-forge
setuptools 40.6.3 py37_0 conda-forge
shiboken2 5.12.0 <pip>
six 1.12.0 py37_0
sqlite 3.26.0 hfa6e2cd_1000 conda-forge
traitlets 4.3.2 py37_0
vc 14.1 h0510ff6_4
vs2015_runtime 14.15.26706 h3a45250_0
wcwidth 0.1.7 py37_0
wheel 0.32.3 py37_0 conda-forge
wincertstore 0.2 py37_1002 conda-forge
zlib 1.2.11 h2fa13f4_1004 conda-forge
There could be a version mismatch issue.
I can see you have PySide2 from conda-forge which is really outdated (5.6),
since shiboken2 is using the latest available in pypi (5.12.0),
additionally the qt you have there is not the same for the PySide2 version (5.6.2).
At the moment there is no conda-forge PySide2 package that uses the latest version, which is 5.12.0.
I guess you can either contribute to the project and update the package, or move back and use virtualenv.
Windows 10
Latest Anaconda3 build
I created a virtual environment using conda.
I activited this environment (BTW, you must use CMD, as it will not switch if you're using powershell)
Now, gunicorn package cannot be installed using conda (unkown package), therefore I've needed to use pip. Pip has installed the package successfully.
When I do "conda list" I see the package:
# packages in environment at C:\Anaconda3\envs\HerokuApp:
#
click 6.6 py35_0
flask 0.11.1 py35_0
gunicorn 19.6.0 <pip>
itsdangerous 0.24 py35_0
jinja2 2.8 py35_1
markupsafe 0.23 py35_2
mkl 11.3.3 1
nltk 3.2.1 py35_0
numpy 1.11.1 py35_0
pandas 0.18.1 np111py35_0
pip 8.1.2 py35_0
python 3.5.1 5
python-dateutil 2.5.3 py35_0
pytz 2016.4 py35_0
scikit-learn 0.17.1 np111py35_1
scipy 0.17.1 np111py35_1
setuptools 23.0.0 py35_0
six 1.10.0 py35_0
vs2015_runtime 14.0.25123 0
werkzeug 0.11.10 py35_0
wheel 0.29.0 py35_0
But when I run "conda list -e" for output to a requirement.txt file (for Heroku), the package is not listed
# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: win-64
click=6.6=py35_0
flask=0.11.1=py35_0
itsdangerous=0.24=py35_0
jinja2=2.8=py35_1
markupsafe=0.23=py35_2
mkl=11.3.3=1
nltk=3.2.1=py35_0
numpy=1.11.1=py35_0
pandas=0.18.1=np111py35_0
pip=8.1.2=py35_0
python=3.5.1=5
python-dateutil=2.5.3=py35_0
pytz=2016.4=py35_0
scikit-learn=0.17.1=np111py35_1
scipy=0.17.1=np111py35_1
setuptools=23.0.0=py35_0
six=1.10.0=py35_0
vs2015_runtime=14.0.25123=0
werkzeug=0.11.10=py35_0
wheel=0.29.0=py35_0
Does anyone know what/why this is?
Thanks
I'm not sure exactly on the what/why but if you use conda env export you'll get the pip installed packages too. Not in the requirement.txt format but it could be useful.
BTW I do see a py3.5 version of gunicorn for windows over here if that helps. It is only version 19.3.0 but you can conda install it via conda install -c phumke gunicorn=19.3.0.