Install Tensorflow from source - installation

I'm trying to install tensorflow on a cluster (I have no sudo rights).
If I try to use pip I first set the environment variable:
export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.9.0-cp27-none-linux_x86_64.whl
and then I execute:
pip install --upgrade $TF_BINARY_URL -b=/home/js668623/tools/tensorflow
The output is:
You are using pip version 7.1.0, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting tensorflow==0.9.0 from https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.9.0-cp27-none-linux_x86_64.whl
Using cached https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.9.0-cp27-none-linux_x86_64.whl
Collecting six>=1.10.0 (from tensorflow==0.9.0)
Using cached six-1.10.0-py2.py3-none-any.whl
Collecting protobuf==3.0.0b2 (from tensorflow==0.9.0)
Using cached protobuf-3.0.0b2-py2.py3-none-any.whl
Collecting wheel (from tensorflow==0.9.0)
Using cached wheel-0.29.0-py2.py3-none-any.whl
Collecting numpy>=1.8.2 (from tensorflow==0.9.0)
Using cached numpy-1.11.1.zip
Collecting setuptools (from protobuf==3.0.0b2->tensorflow==0.9.0)
Using cached setuptools-23.1.0-py2.py3-none-any.whl
Installing collected packages: six, setuptools, protobuf, wheel, numpy, tensorflow
Found existing installation: six 1.9.0
Uninstalling six-1.9.0:
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/pip/basecommand.py", line 223, in main
status = self.run(options, args)
File "/usr/lib/python2.7/site-packages/pip/commands/install.py", line 308, in run
strip_file_prefix=options.strip_file_prefix,
File "/usr/lib/python2.7/site-packages/pip/req/req_set.py", line 640, in install
requirement.uninstall(auto_confirm=True)
File "/usr/lib/python2.7/site-packages/pip/req/req_install.py", line 726, in uninstall
paths_to_remove.remove(auto_confirm)
File "/usr/lib/python2.7/site-packages/pip/req/req_uninstall.py", line 125, in remove
renames(path, new_path)
File "/usr/lib/python2.7/site-packages/pip/utils/__init__.py", line 314, in renames
shutil.move(old, new)
File "/usr/lib64/python2.7/shutil.py", line 299, in move
rmtree(src)
File "/usr/lib64/python2.7/shutil.py", line 252, in rmtree
onerror(os.remove, fullname, sys.exc_info())
File "/usr/lib64/python2.7/shutil.py", line 250, in rmtree
os.remove(fullname)
OSError: [Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/six-1.9.0-py2.7.egg-info/PKG-INFO'

If you don't have sudo rights on the machine, personally I find the "virtualenv" installation easiest. It should just work without root access.
https://www.tensorflow.org/versions/r0.9/get_started/os_setup.html#virtualenv-installation
You might also look into the "anaconda" installation, but I have no experience with that.

If you also need a specific version of python, you 'll need more than the script I give you, but with a little bit of diggig, that script should work for you ;)
module load Python/2.7.11-foss-2016a
curl -O https://bootstrap.pypa.io/get-pip.py
python get-pip.py --user --upgrade
rm get-pip.py
.local/bin/pip install --user --upgrade pip
git clone https://github.com/tensorflow/tensorflow
git clone https://github.com/bazelbuild/bazel.git
wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u92-b14/jdk-8u92-linux-x64.tar.gz
tar -xf jdk*
rm jdk*.gz
cd bazel
export JAVA_HOME=~/jdk1.8.0_92/
module unload GCC/4.9.3-2.25
module load GCC/4.8.2
./compile.sh
cd ../tensorflow
export PATH=/home/ulg/sysmod/gmath/bazel/output/:$PATH
./configure
bazel build -c opt //tensorflow/tools/pip_package:build_pip_package 2>/dev/null
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
cd
.local/bin/pip install --user --upgrade /tmp/tensorflow_pkg/tensorflow-0.8.0-py2-none-any.whl

Related

Install PANDAS on Mac, big issue

I lot of people had asked this question, but I can't find an answer that can help me overcome my problems installing PANDAS on my Mac.
I've tried several procedures previously suggested, but they don't work. This is the error I'm getting.
Pablos-MacBook-Pro:pastudilloe$ sudo pip install pandas
The directory '/Users/pastudilloe/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/pastudilloe/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting pandas
Downloading https://files.pythonhosted.org/packages/58/9a/f571a032f5f2bf5b8a5d63ce079428c5423299853ed2d8d96626f441801d/pandas-0.23.3-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (14.6MB)
100% |████████████████████████████████| 14.6MB 83kB/s
Collecting numpy>=1.9.0 (from pandas)
Downloading https://files.pythonhosted.org/packages/3c/bf/e36756c562f7386be78c6942f0a8a647ee4eb374cdf219bece7054832b14/numpy-1.15.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (24.5MB)
100% |████████████████████████████████| 24.5MB 49kB/s
Collecting python-dateutil>=2.5.0 (from pandas)
Downloading https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl (211kB)
100% |████████████████████████████████| 215kB 640kB/s
Requirement already satisfied: pytz>=2011k in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python (from pandas)
Requirement already satisfied: six>=1.5 in /Users/pastudilloe/Library/Python/2.7/lib/python/site-packages (from python-dateutil>=2.5.0->pandas)
Installing collected packages: numpy, python-dateutil, pandas
Found existing installation: numpy 1.8.0rc1
DEPRECATION: Uninstalling a distutils installed project (numpy) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling numpy-1.8.0rc1:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 778, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 754, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 267, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-62NbGp-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy-1.8.0rc1-py2.7.egg-info'
You are using pip version 9.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
uninstall numpy and then try again:
pip uninstall numpy
pip install pandas
In the ipython jupyter code,
Add below lines
!pip install pandas
!pip install numpy
It will be installed during runtime if it is not installed.

"OSError: mysql_config not found" when trying to "pip install mysqlclient" - Django

I'm relatively new to Django, and Web Development in general.
I am trying to
pip install mysqlclient
in my
virtualenv -p python3
to hook up Django 2.0 to mySQL. However, I'm getting this error:
Collecting mysqlclient
Using cached mysqlclient-1.3.12.tar.gz
Complete output from command python setup.py egg_info:
/bin/sh: mysql_config: command not found
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/43/md5vpqrx0mx8627sq04slbz00000gn/T/pip-build-l8ea3vja/mysqlclient/setup.py", line 17, in <module>
metadata, options = get_config()
File "/private/var/folders/43/md5vpqrx0mx8627sq04slbz00000gn/T/pip-build-l8ea3vja/mysqlclient/setup_posix.py", line 44, in get_config
libs = mysql_config("libs_r")
File "/private/var/folders/43/md5vpqrx0mx8627sq04slbz00000gn/T/pip-build-l8ea3vja/mysqlclient/setup_posix.py", line 26, in mysql_config
raise EnvironmentError("%s not found" % (mysql_config.path,))
OSError: mysql_config not found
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/43/md5vpqrx0mx8627sq04slbz00000gn/T/pip-build-l8ea3vja/mysqlclient/
I have looked around for answers for hours, and yes, I have checked out this and this but nothing seems to be working.
Any help would be much appreciated!
the python package "mysqlclient" is depended on C system library to drive mysql connect,if you on ubuntu 16.4, just do as below:
sudo apt-get install libmysqlclient-dev
sudo pip install mysqlclient
If you get OSError: mysql_config not found when installing the mysqlclient Python package in macOS, then just do:
brew install mysql-client
export PATH="/usr/local/opt/mysql-client/bin:$PATH"
and retry.
You should start with cloning the mysqlclient repo:
git clone https://github.com/PyMySQL/mysqlclient-python
Then you will have to get mysql-connector-c. You can get it by doing:
brew install mysql-connector-c
Then open /usr/local/bin/mysql_config in a text editor.
From the Github issue:
There are lines in mysql_config like following:
# Create options
libs="-L$pkglibdir"
libs="$libs -l "
It should be:
#Create options
libs="-L$pkglibdir"
libs="$libs -lmysqlclient -lssl -lcrypto"
Save that file and change back to the directory where you downloaded the repo. Now open site.cfg file.
uncomment the line (remove the #)
#mysql_config = /usr/local/bin/mysql_config
Save that and run:
python3 setup.py install
For CentOS here's what made it work for me:
yum install gcc mysql-devel openssl-devel
#Install library explicitly, otherwise the python3 setup.py install will fail
pip3 install "mysqlclient==2.0.3" --global-option=build_ext --global-option="-L/usr/lib64/mysql/"

How can I install Ansible on Raspbian?

I would like to install Ansible on my Raspberry Pi (raspbian/jessie) so that I can maintain it. Unfortunately, I don't seem to be able to install it. I've tried APT and PIP so far but each has a hurdle.
APT
pi#retropie:~ $ sudo apt-add-repository ppa:ansible/ansible
Traceback (most recent call last):
File "/usr/bin/apt-add-repository", line 167, in <module>
sp = SoftwareProperties(options=options)
File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py", line 105, in __init__
self.reload_sourceslist()
File "/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py", line 595, in reload_sourceslist
self.distro.get_sources(self.sourceslist)
File "/usr/lib/python3/dist-packages/aptsources/distro.py", line 89, in get_sources
(self.id, self.codename))
aptsources.distro.NoDistroTemplateException: Error: could not find a distribution template for Raspbian/jessie
PIP (Python 2.7.9, pip 1.5.6)
pi#retropie:~ $ sudo pip install ansible
Downloading/unpacking ansible
Downloading ansible-2.4.1.0.tar.gz (6.7MB): 6.7MB downloaded
no previously-included directories found matching 'ticket_stubs'
no previously-included directories found matching 'hacking'
Downloading/unpacking jinja2 (from ansible)
Downloading Jinja2-2.9.6-py2.py3-none-any.whl (340kB): 340kB downloaded
Downloading/unpacking PyYAML (from ansible)
Downloading PyYAML-3.12.tar.gz (253kB): 253kB downloaded
Running setup.py (path:/tmp/pip-build-bsUTB2/PyYAML/setup.py) egg_info for package PyYAML
Downloading/unpacking paramiko (from ansible)
Downloading paramiko-2.3.1-py2.py3-none-any.whl (182kB): 182kB downloaded
Downloading/unpacking cryptography (from ansible)
Downloading cryptography-2.1.2.tar.gz (441kB): 441kB downloaded
Running setup.py (path:/tmp/pip-build-bsUTB2/cryptography/setup.py) egg_info for package cryptography
error in cryptography setup command: Invalid environment marker: python_version < '3'
Complete output from command python setup.py egg_info:
error in cryptography setup command: Invalid environment marker: python_version < '3'
----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip-build-bsUTB2/cryptography
Storing debug log for failure in /root/.pip/pip.log
Python 3
As you can see from the above error, "cryptography" indicates it needs > python3
I looked at the installation requirements and found that you need python 3.5+ if you use 3. Rasbian comes with 3.4...
I then tried to get python 3.6 installed using a couple of ppa's (ppa:jonathonf/python-3.6 and ppa:deadsnakes/ppa) but there aren't distros available for raspbian/jessie either.
After that I started looking at pulling down and compiling python from source but I'm expecting I'll hit another hurdle.
Ugh...
In summary, does anyone have any ideas how I can get Ansible installed on a Raspberry Pi?
Slightly old question but its the first result that comes up when googling for how to install Ansible on Raspbian so thought I'd update it. If you're running stretch (or stretch-lite)
You can just do
sudo apt-get install ansible
However currently the version of Ansbile in the Raspbian repositories is 2.2 which is a little old
Following instructions (with a little modification to overcome some errors) from the Ansible installation page you can do the following:
First run
sudo apt-get install dirmngr
Edit your /etc/apt/sources.list and append
deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main
then run
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 93C4A3FD7BB9C367
sudo apt-get update
sudo apt-get install ansible
This will give you version 2.7.8 as of today.
Edited to include missing command (thanks SpacePope) and correct formatting.
Jessie was released in 2015 and is officially obsolete. Stretch is the current Raspbian repo, and it has python3.5 without adding PPAs.
You can then simply install Ansible with pip3.

How can I resolve this issue regarding upgradation of pip on ubuntu 16.04?

Here is what happened.I even tried to remove pip, but it shows the same problem.My python version is 2.7.12
saikat#saikat:~$ pip install --upgrade pip
Collecting pip
Using cached pip-9.0.1-py2.py3-none-any.whl
Installing collected packages: pip
Found existing installation: pip 8.1.2
Uninstalling pip-8.1.2:
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 317, in run
prefix=options.prefix_path,
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 736, in install
requirement.uninstall(auto_confirm=True)
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 742, in uninstall
paths_to_remove.remove(auto_confirm)
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/usr/local/lib/python2.7/dist-packages/pip/utils/__init__.py", line 267, in renames
shutil.move(old, new)
File "/usr/lib/python2.7/shutil.py", line 303, in move
os.unlink(src)
OSError: [Errno 13] Permission denied: '/usr/bin/pip'
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
try this.
wget https://pypi.python.org/packages/e7/a8/7556133689add8d1a54c0b14aeff0acb03c64707ce100ecd53934da1aa13/pip-8.1.2.tar.gz
tar -xzvf pip-8.1.2.tar.gz
cd pip-8.1.2
sudo python setup.py install
It should help

Problems in using pip to install tensorflow on iMac El Capitan?

I've followed the installation guide but the final install step failed.
My installed python version:
python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
I have used the following two commands to install tensorflow
sudo pip install --upgrade $TF_BINARY_URL
sudo -H pip install --upgrade $TF_BINARY_UR
Somehow I found Protobuf was not installed under my Python 2.7,
so I followed the suggestion from the installation site,
pip install protobuf
It failed to install protobuf, see the error message below.
Please help
Thanks in advance
When using the first command, got the following messages:
The directory '/Users/David/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/David/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Then as suggested, I used the second command, then error messages were:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip-8.1.2-py2.7.egg/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip-8.1.2-py2.7.egg/pip/commands/install.py", line 317, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip-8.1.2-py2.7.egg/pip/req/req_set.py", line 736, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip-8.1.2-py2.7.egg/pip/req/req_install.py", line 742, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip-8.1.2-py2.7.egg/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip-8.1.2-py2.7.egg/pip/utils/__init__.py", line 267, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 299, in move
copytree(src, real_dst, symlinks=True)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 208, in copytree
raise Error, errors
Error: [('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', '/tmp/pip-9CKE1y-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py', "[Errno 1] Operation not permitted: '/tmp/pip-9CKE1y-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', '/tmp/pip-9CKE1y-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-9CKE1y-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/__init__.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', '/tmp/pip-9CKE1y-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py', "[Errno 1] Operation not permitted: '/tmp/pip-9CKE1y-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.py'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', '/tmp/pip-9CKE1y-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc', "[Errno 1] Operation not permitted: '/tmp/pip-9CKE1y-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib/markers.pyc'"), ('/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', '/tmp/pip-9CKE1y-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib', "[Errno 1] Operation not permitted: '/tmp/pip-9CKE1y-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/_markerlib'")]
For the command 3, the error message:
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/private/var/folders/h5/vh7y894j0js3vqkv_hz4qk6w0000gn/T/pip-build-KWqDAq/protobuf/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/h5/vh7y894j0js3vqkv_hz4qk6w0000gn/T/pip-5RoEnu-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/h5/vh7y894j0js3vqkv_hz4qk6w0000gn/T/pip-build-KWqDAq/protobuf/
I think this is a result of System Integrity Protection in El Capitan. I was able to install using the following:
# Note the lack of "sudo" and the addition of "--user"
pip install --user --upgrade $TF_BINARY_URL
# And then to ensure that the user-specific packages are used,
# I had to start python from the user-site packages directory
cd $(python -m site --user-site)
I had a similar problem when installing awsebcli,
try:
sudo pip install protobuf --ignore-installed six

Resources