everyone. I'm try to used Dask with Distributed + HDFS for processing some files. when I installed the distributed try to install the HDFS3 plugins, the error was :
Can not find the shared library:libhdfs3.so
My environment is Ubuntu 16 Desktop version. I strict according to bewlo, but still not working. Hope someone can help! Thanks a lot
conda install hdfs3 -c conda-forge
echo "deb https://dl.bintray.com/wangzw/deb trusty contrib" | sudo tee /etc/apt/sources.list.d/bintray-wangzw-deb.list
sudo apt-get install -y apt-transport-https
sudo apt-get update
sudo apt-get install libhdfs3 libhdfs3-dev
pip install hdfs3
If you are not using conda and you are using pip, you can also face this error
Can not find the shared library:libhdfs3.so
To ensure requirements in this page https://github.com/ContinuumIO/libhdfs3-downstream/tree/master/libhdfs3
solved my problem.
There are packages that you should install, if you are installing with pip:
cmake (2.8+) http://www.cmake.org/
boost (tested on 1.53+) http://www.boost.org/
google protobuf http://code.google.com/p/protobuf/
libxml2 http://www.xmlsoft.org/
kerberos http://web.mit.edu/kerberos/
libuuid http://sourceforge.net/projects/libuuid/
libgsasl http://www.gnu.org/software/gsasl/ (need https://github.com/bdrosen96/libgsasl)
openssl https://www.openssl.org/
P.S.: OS: Centos7
Since no one answers this question and I figure out the solution by myself , at least it works for me.
conda install libhdfs3
pip install libhdfs3
conda install -c clinicalgraphics libgcrypt11
conda install libprotobuf=2.5
conda update libhdfs3
if still not work try to update.
Related
After many attempts figuring out how to install PyBluez on Ubuntu using pip3 I've seen many people with the same problem. fortunatelly I found the way to do it without pip3:
downloaded the program from https://pybluez.readthedocs.io/en/latest/
then I installed this library using "sudo apt-get install libbluetooth-dev"
and finally installed PyBluez by running "pyhton3 setup.py install" in the folder that I previously downloaded.
My question is why I cant use pip3 to install PyBluez?
I tried sudo pip3 install PyBluez :( :(
I am a newbie to Airflow. i have some trouble to remove Airflow v1.10.3 ,i am using pip3 version 8.1.1 on Ubuntu 16.04.
I already tried to remove pip with sudo apt-get remove python3-pip
and sudo apt-get remove pip3 and all his dependencies.
and tried to remove all libraries related with Python.
But i stil have Airflow and his commands down in terminal.
If you are not able to uninstall with
pip uninstall airflow
Maybe the reason is for the latest versions of Airflow this command will work
pip uninstall apache-airflow
Credits to This Answer.
The command apt-get remove pip doesn't remove pip-installed libraries.To uninstall pip-installed libraries you need to install pip back and then uninstall the libraries using pip:
sudo apt install pip # or pip3
pip uninstall airflow
or
pip3 uninstall airflow
depending on what python version you use.
I'm used to tortoise SVN. But after switching to centos 7 it is very difficult to use it without GUI. Please Help me install RabbitVCS instead.
The below instructions worked for me in May 2019. YMMV.
Install EPEL Release
export http_proxy=http://{proxyhost}:{port}
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install epel-release-latest-7.noarch.rpm
Install Pre-requisites
sudo yum install meld pysvn python-dulwich python-simplejson
Fetch RabbitVCS packages
export ftp_proxy=$http_proxy
wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/hmandevteam/CentOS_7/noarch/rabbitvcs-core-0.17.1-3.2.noarch.rpm
wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/hmandevteam/CentOS_7/noarch/rabbitvcs-cli-0.17.1-3.2.noarch.rpm
wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/hmandevteam/CentOS_7/noarch/python2-rabbitvcs-0.17.1-3.2.noarch.rpm
wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/hmandevteam/CentOS_7/noarch/rabbitvcs-nautilus-0.17.1-3.2.noarch.rpm
Install RabbitVCS packages
sudo yum install rabbitvcs-core-0.17.1-3.2.noarch.rpm
sudo yum install rabbitvcs-cli-0.17.1-3.2.noarch.rpm
sudo yum install python2-rabbitvcs-0.17.1-3.2.noarch.rpm
sudo yum install rabbitvcs-nautilus-0.17.1-3.2.noarch.rpm
Reboot
sudo reboot
Open File Manager and you can see RabbitVCS Menu
If the above rpm ftp locations become stale, goto http://rpm.pbone.net and search for the following packages with Centos 7 filter -
rabbitvcs-core
rabbitvcs-cli
python2-rabbitvcs
rabbitvcs-nautilus
You may find these packages at other locations too
NOTE: You may sometimes require these pre-requisite packages too pygtk, python-configobj, python-gobject, python-gtkspell, python-svn
I had same problem when working with centos7, ended up using thunar instead of nautilus. See stackexchange and github issue.
Update:
I have successfully build rabbitVCS following steps from the github.com and then copied the nautilus extension.
I am trying to run some python script using ssh to log into the google compute engine but all the installed pip modules are not found as I do not have permission to the .cache/pip folder in my user is there a correct way to do this?
You should be running this with the root user.
Also, if you need pip inside your GCP Instance, you can use the following commands:
sudo curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
sudo python get-pip.py
[Source]
Use:
sudo apt-get install python3-pip
sudo runs this command as an administrator
apt-get is the standard package manager used on Debian Linux distributions
python3-pip is the package name for pip3
Once installed, you can install PIP modules with:
pip3 install MODULE_NAME
for example:
pip3 install tensorflow
I'm not entirely sure there is one correct way to do this, but an easy way would be to use the conda python package manager.
The lighter version of it is miniconda. You can get a minimal python installation with pip preinstalled, and virtual environments capability if you need. Assuming you are running on linux and want python 3, you'll have to run
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
and then install conda with
bash Miniconda3-latest-Linux-x86_64.sh
At the end of this process you should have a minimal python installation (that includes pip) and you'll be able to install packages with pip as you are used to.
You might want to install some basic libraries first -
sudo apt-get install bzip2 libxml2-dev
Then install miniconda as given by #teoguso and restart your shell
source ~/.bashrc
You can then use conda or pip to install your packages
CentOS 7 EPEL now includes Python 3.4: yum install python34
However, when I try that, even though Python 3.4 installs successfully, it doesn't appear to install pip. Which is weird, because pip should be included by default with Python 3.4. which pip3 doesn't find anything, nor does which pip.
How do I access pip from the Python 3.4 package in CentOS 7 EPEL release?
The easiest way I've found to install pip3 (for python3.x packages) on CentOS 7 is:
$ sudo yum install python34-setuptools
$ sudo easy_install-3.4 pip
You'll need to have the EPEL repository enabled before hand, of course.
You should now be able to run commands like the following to install packages for python3.x:
$ pip3 install foo
curl https://bootstrap.pypa.io/get-pip.py | python3.4
Or if you don't have curl for some reason:
wget https://bootstrap.pypa.io/get-pip.py
python3.4 get-pip.py
After this you should be able to run
$ pip3
The CentOS 7 yum package for python34 does include the ensurepip module, but for some reason is missing the setuptools and pip files that should be a part of that module. To fix, download the latest wheels from PyPI into the module's _bundled directory (/lib64/python3.4/ensurepip/_bundled/):
setuptools-18.4-py2.py3-none-any.whl
pip-7.1.2-py2.py3-none-any.whl
then edit __init__.py to match the downloaded versions:
_SETUPTOOLS_VERSION = "18.4"
_PIP_VERSION = "7.1.2"
after which python3.4 -m ensurepip works as intended. Ensurepip is invoked automatically every time you create a virtual environment, for example:
pyvenv-3.4 py3
source py3/bin/activate
Hopefully RH will fix the broken Python3.4 yum package so that manual patching isn't needed.
Update: The python34 bug mentioned below has finally been fixed. It is a perfectly fine choice now.
Rather than using broken EPEL python34 packages, you can enable the IUS repo and have it work properly.
pip inside virtual environments
The main python34u and python35u IUS packages include the pyvenv tool (/usr/bin/pyvenv-3.4 or /usr/bin/pyvenv-3.5) that includes bundled wheels of pip and setuptools for bootstrapping virtual environments.
global pip
The python34u-pip and python35u-pip IUS packages include /usr/bin/pip3.4 and /usr/bin/pip3.5 respectively. These work just fine to install packages to the system site-packages directory.
yum install python34-pip
pip3.4 install foo
You will likely need the EPEL repositories installed:
yum install -y epel-release
Update 2019
I tried easy_install at first but it doesn't install packages in a clean and intuitive way. Also when it comes time to remove packages it left a lot of artifacts that needed to be cleaned up.
sudo yum install epel-release
sudo yum install python34-pip
pip install package
Was the solution that worked for me, it installs "pip3" as pip on the system. It also uses standard rpm structure so it clean in its removal. I am not sure what process you would need to take if you want both python2 and python3 package manager on your system.
Below are the steps I followed to install python34 and pip
yum update -y
yum -y install yum-utils
yum -y groupinstall development
yum -y install https://centos7.iuscommunity.org/ius-release.rpm
yum makecache
yum -y install python34u python34u-pip
python3.6 -v
echo "alias python=/usr/bin/python3.4" >> ~/.bash_profile
source ~/.bash_profile
pip3 install --upgrade pip
# if yum install python34u-pip doesnt work, try
curl https://bootstrap.pypa.io/get-pip.py | python
There is a easy way of doing this by just using easy_install (A Setuptools to package python librarie).
Assumption.
Before doing this check whether you have python installed into your Centos machine (at least 2.x).
Steps to install pip.
So lets do install easy_install,
sudo yum install python-setuptools python-setuptools-devel
Now lets do pip with easy_install,
sudo easy_install pip
That's Great. Now you have pip :)
Figure out what version of python3 you have installed:
yum search pip
and then install the best match. Use reqoquery to find name of resulting pip3.e.g
repoquery -l python36u-pip
tells me to use pip3.6 instead of pip3
On CentOS 7, the pip version is pip3.4 and is located here:
/usr/local/bin/pip3.4