I have an PyQt5 project on github, and the following is my environment settings:
dist: trusty
sudo: required
language: python
python:
- 3.4
- 3.5
before_install:
#SIP
- cd ..
- curl -L -O "https://sourceforge.net/projects/pyqt/files/sip/sip-4.19.3/sip-4.19.3.tar.gz"
- tar -xvf sip-4.19.3.tar.gz
- cd sip-4.19.3
- python configure.py
- sudo make install
#Qt5
- sudo add-apt-repository -y "ppa:beineri/opt-qt59-trusty"
- sudo apt-get update -qq
- sudo apt-get install qt59-meta-full qt59charts-no-lgpl
- QTDIR="/opt/qt59"
- PATH="$QTDIR/bin:$PATH"
- source /opt/qt59/bin/qt59-env.sh
- qmake -v
#PyQt5
- cd ..
- curl -L -O "https://sourceforge.net/projects/pyqt/files/PyQt5/PyQt-5.9/PyQt5_gpl-5.9.tar.gz"
- tar -xvf PyQt5_gpl-5.9.tar.gz
- cd PyQt5_gpl-5.9
- python configure.py --confirm-license
- sudo make install
#QScintilla
- cd ..
- curl -L -O "https://sourceforge.net/projects/pyqt/files/QScintilla2/QScintilla-2.10.1/QScintilla_gpl-2.10.1.tar.gz"
- tar -xvf QScintilla_gpl-2.10.1.tar.gz
- cd QScintilla_gpl-2.10.1
- cd Qt4Qt5
- qmake qscintilla.pro
- sudo make install
- cd ../designer-Qt4Qt5
- qmake designer.pro
- sudo make install
- cd ../Python
- python configure.py --pyqt=PyQt5
- sudo make install
#PyQtChart
- cd ../..
- curl -L -O "https://sourceforge.net/projects/pyqt/files/PyQtChart/PyQtChart-5.9/PyQtChart_gpl-5.9.tar.gz"
- tar -xvf PyQtChart_gpl-5.9.tar.gz
- cd PyQtChart_gpl-5.9
- python configure.py
- sudo make install
- cd $TRAVIS_BUILD_DIR
install:
- pip install -r requirements.txt
- pip install pyinstaller
script:
- make
before_cache:
- rm -rf $HOME/.cache/pip/log
cache:
directories:
- $HOME/.cache/pip
My project requires SIP, PyQt 5.7 or higher (for PyQtChart), QScintilla and some python modules.
One can imagine that it has been more than 50 minutes.
Is there any way to make these steps faster?
This is more of a workaround than a proper fix, but I've been using miniconda to successfully install pyqt for me. This isn't super fast but is a lot faster than what you're currently doing.
The following lines will get you into a python virtual environment with pyqt installed:
wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh;
bash ./miniconda.sh -b -p ${HOME}/miniconda;
export PATH=${HOME}/miniconda/bin:$PATH;
conda install --yes python="2.7" pyqt;
Another solution could be using the unofficial PyPI exposed and maintained here: https://github.com/pyqt/python-qt5
By running pip install python-qt5. Keep in mind this is not official and may break/disfunction. Use at your own risk.
Additionally, keep in mind that with python 3.5 and up, you can install pyqt5 using pip by simply running
pip3 install pyqt5
Last but not least, Travis' maintainers may be willing to add pyqt builds as part of their own environments, either pre-installed or available as precompiled binaries for installation using pip.
I'll +1 any such request.
Related
Executing this script:
sudo yum install -y gcc zlib openssl-devel bzip2-devel libffi-devel zlib-devel xz-devel
cd /tmp/
sudo wget --quiet https://www.python.org/ftp/python/3.7.13/Python-3.7.13.tgz
sudo tar xzf Python-3.7.13.tgz
cd Python-3.7.13
sudo ./configure --enable-optimizations --prefix=/usr
sudo make altinstall
sudo rm /tmp/Python-3.7.13.tgz
Python3.7 installs but not pip.
$ pip3.7 -V
Traceback (most recent call last):
File "/usr/local/bin/pip3.7", line 5, in <module>
from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip._internal'
Also tried
$ python3.7 -m pip install pip
/usr/local/bin/python3.7: No module named pip.__main__; 'pip' is a package and cannot be directly executed
When I try running sudo python3.7 it also says it's not found. I think the installing of python is just messed up and all over the place. The current machine has python 2.7.5 and 3.6.8. Also, python37 is install in /usr/local/lib rather than /usr/bin.
EDIT 1 - Adding --prefix=/usr to the end of sudo ./configure --enable-optimizations fixed the 2 strikethrough issues. It's installing the the correct spots and I'm seeing pip3.7 being installed in /usr/bin but I'm still getting the same error.
EDIT 2 -
$ sudo python3.7 -m ensurepip
Looking in links: /tmp/tmpjsylb056
Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (47.1.0)
Requirement already satisfied: pip in /usr/lib/python3.7/site-packages (22.1.2)
Edit 3 Fixed? - Adding sudo chmod 755 /usr/lib/python3.7/site-packages/ -R after sudo make altinstall made me use pip3.7 without sudo. Doubt this is the correct approach for this...
Getting SSL Certificate error while deploying the following code on aws lambda using aws codestar build pipeline.
Looked at multiple community discussions, nothing worked out.
version: 0.2
phases:
install:
commands:
# Upgrade AWS CLI & PIP to the latest version
- pip install --upgrade awscli
- pip install --upgrade pip
# Define Directories
- export HOME_DIR=`pwd`
- export NLTK_DATA=$HOME_DIR/nltk_data
pre_build:
commands:
- cd $HOME_DIR
# Create VirtualEnv to package for lambda
- virtualenv venv
- . venv/bin/activate
# Install Supporting Libraries
- pip install -U scikit-learn
- pip install -U requests
# Install WordNet
- pip install -U nltk
- python -m nltk.downloader -d $NLTK_DATA punkt
# Output Requirements
- pip freeze > requirements.txt
# Discover and run unit tests in the 'tests' directory. For more information, see <https://docs.python.org/3/library/unittest.html#test-discovery>
- python -m unittest discover tests
build:
commands:
- cd $HOME_DIR
- mv $VIRTUAL_ENV/lib/python3.6/site-packages/* .
Only way that worked for me was download the modules and install them into my source folder in a nltk_data folder then create a lambda environment variable NLTK with value ./nltk_data
Is it possible to use travis-ci to build a c++ application/project that uses cmake, gcc-6 and g++-6?
Configuring travis to use the right compiler is a bit tricky.
This is how it can be done:
First of all you need to set the distribution to trusty (the newest version of ubuntu that's supported by travis-ci) and require sudo.
dist: trusty
sudo: require
Next up we set the language and the compiler:
language: cpp
compiler: gcc
So far so good. Now we can go about setting up the apt install configuration:
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- gcc-6
- g++-6
- cmake
This adds the ppa for the newer version of our build tools and installs them. The next step is setting up links to the new gcc and g++. /usr/local/bin is being searched before /usr/bin, so that our newly installed version 6 compilers are going to be accessible with just gcc and g++. The beginning of your script: should look like this:
script:
- sudo ln -s /usr/bin/gcc-6 /usr/local/bin/gcc
- sudo ln -s /usr/bin/g++-6 /usr/local/bin/g++
Add the next line as well, if you want to verify the versions of those tools:
- gcc -v && g++ -v && cmake --version
The versions that come back from these commands are as follows:
gcc: 6.2.0
g++: 6.2.0
cmake: 3.2.2
That's basically it. The complete .travis.yml looks like this:
dist: trusty
sudo: required
language:
- cpp
compiler:
- gcc
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- gcc-6
- g++-6
- cmake
script:
# Link gcc-6 and g++-6 to their standard commands
- ln -s /usr/bin/gcc-6 /usr/local/bin/gcc
- ln -s /usr/bin/g++-6 /usr/local/bin/g++
# Export CC and CXX to tell cmake which compiler to use
- export CC=/usr/bin/gcc-6
- export CXX=/usr/bin/g++-6
# Check versions of gcc, g++ and cmake
- gcc -v && g++ -v && cmake --version
# Run your build commands next
I found some errors in #henne90gen's answer (or maybe they've just changed). Specifically:
You don't need sudo.
You don't need to install CMake from apt. This will install an ancient CMake 2.8 from Trusty. Fortunately the build image already comes with CMake 3.9.2 (as of now).
gcc-7 doesn't get installed to /usr/local/bin and it is already in the PATH.
This should work:
dist: trusty
language: cpp
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- gcc-7
- g++-7
script:
- export CC=gcc-7
- export CXX=g++-7
- ...
Here's a longer example that includes a modern version of Qt (with QtSVG which I'm using), and works on OSX and Linux.
os:
- linux
- osx
language: cpp
dist: trusty
addons:
apt:
sources:
- ubuntu-toolchain-r-test
- sourceline: "ppa:beineri/opt-qt-5.10.1-trusty"
packages:
- gcc-7
- g++-7
- qt510-meta-minimal
- qt510svg
- qt510imageformats
- qt510tools
before_install:
- if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
brew update ;
brew install qt5 cmake ;
brew link --force qt ;
fi
script:
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then
export CC=gcc-7 ;
export CXX=g++-7 ;
source /opt/qt510/bin/qt510-env.sh ;
fi
- cmake --version
- qmake --version
- ...
Use dist: bionic. This should meet most of the cases.
Adding a semi-related solution after struggling with this for way too long. Hopefully, it helps someone else avoid spending the amount of time I did going through the cycle of updating travis.yml, committing, waiting for Travis...repeat til it works.
I had a C extension in a Python project that started producing failing tests in Travis, yet passed locally. Eventually, I tracked it down to the old version of gcc on the xenial environment. Here's the Travis YAML file that finally solved the issue for me:
dist: xenial
language: python
before_install:
- sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
- sudo apt-get update -q
- sudo apt-get install -y gcc-7
- export CC=/usr/bin/gcc-7
python:
- "3.6"
- "3.7"
install:
- pip install -r requirements.txt
- python setup.py build_ext --inplace
script:
- python run_tests.py
As an aside, anyone else find they want to trigger a Travis build on a specific commit to find out exactly where a problem started?
(py36venv) vagrant#pvagrant-dev-vm:/vagrant/venvs$ pip3 install pep8
pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Collecting pep8 Could not fetch URL
https://pypi.python.org/simple/pep8/: There was a problem confirming
the ssl certificate: Can't connect to HTTPS URL because the SSL module
is not available. - skipping
Could not find a version that satisfies the requirement pep8 (from
versions: ) No matching distribution found for pep8
Background information - Trying to move to python 3.6.
Installed python3.6 using the below commands:
wget https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tgz
tar -xvf Python-3.6.0.tgz
cd Python-3.6.0
./configure --enable-optimizations
make -j8 sudo
make altinstall python3.6
Created virtualenv by:
python3.6 -m venv py36venv
source py36venv/bin/activate
Tried to install pep8
(py36venv) pip3 install pep8
pip is configured with locations that require TLS/SSL, however the ssl
module in Python is not available.
Collecting pep8
Could not fetch URL https://pypi.python.org/simple/pep8/: There was a problem
confirming the ssl certificate: Can't connect to HTTPS URL because the
SSL module is not available. - skipping Could not find a version
that satisfies the requirement pep8 (from versions: ) No matching
distribution found for pep8
I followed the below steps for python3.6 installation in ubuntu 14.04 and virtualenv pip installs works fine.
Python 3.6 Installation:
sudo apt-get install python3-dev libffi-dev libssl-dev
wget https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tgz
tar xvf Python-3.6.0.tgz
cd Python-3.6.0
./configure --enable-optimizations
make -j8
sudo make altinstall
python3.6
If seeing the following error --
zipimport.ZipImportError: can't decompress data; zlib not available
make: *** [altinstall] Error 1
try:
sudo apt-get install zlib1g-dev
Validation:
Create virtualenv in python3.6:
python3.6 -m venv testenv
source testenv/bin/activate
pip install pep8
using pip:
(testenv) vagrant#pvagrant-dev-vm:~$ pip install pep8
*Collecting pep8
Downloading pep8-1.7.0-py2.py3-none-any.whl (41kB)
100% |████████████████████████████████| 51kB 4.1MB/s
Installing collected packages: pep8
Successfully installed pep8-1.7.0*
(testenv) vagrant#pvagrant-dev-vm:~$ pip list
pep8 (1.7.0)
pip (9.0.1)
setuptools (28.8.0)
I stumbled upon the same issue when I tried to create a virtual environment utilising python3.6.0. Here is my solution for Mac OS X 10.12.2 (Py_minion comment was pretty close):
Setup
I created the environment by the following steps:
downloading python3.6.0
running
./configure --prefix=<some_path>`
make
make install
mkvirtualenv --python=<some_path/bin/python3.6> foo
So basically similar to: https://stackoverflow.com/a/11301911/1286093
An indication if you have the same issue as I had would be a similar line when running make
The necessary bits to build these optional modules were not found: _ssl
Solution
Install openssl
brew install openssl
brew unlink openssl && brew link openssl --force
Change Module/Setup or Module/Setup.dist
You can find those files in the directory of the downloaded Python version.
Comment in and, if necessary change, lines 209 - 211 (I had to change the SSL variable to my openssl location).
SSL=/usr/local/opt/openssl <---- THIS DEPENDS ON YOUR INSTALLATION
_ssl _ssl.c \
-DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
-L$(SSL)/lib -lssl -lcrypto
Given that this was the location of openssl
Set environment variables
export CFLAGS="-I$(brew --prefix openssl)/include"
export LDFLAGS="-L$(brew --prefix openssl)/lib"
make and install again
Running
./configure --prefix=<some_path>`
make
make install
mkvirtualenv --python=<some_path/bin/python3.6> foo
again did the trick for me
Running make reported to me in the shell output:
The necessary bits to build these optional modules were not found:
_bz2 _dbm _gdbm
_sqlite3 _ssl _tkinter
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
What solved the problem in my case (Linux Mint 18.1, openssl already installed) was editing the setup.py in the Python-3.6.0 folder adding there the path to where the openssl installation put the ssl.h file on my system into ( /usr/include/openssl/ssl.h ). Here the section in which I have added the line '/usr/include':
# Detect SSL support for the socket module (via _ssl)
search_for_ssl_incs_in = [
'/usr/local/ssl/include/',
'/usr/contrib/ssl/include/',
'/usr/include/'
]
ssl_incs = find_file('openssl/ssl.h', inc_dirs,
search_for_ssl_incs_in
)
I have solved this problem on Ubuntu-16.04.1.
First you need to install necessary libraries. To install open Terminal (Ctrl+Alt+T), then type;
sudo apt-get install build-essential checkinstall
sudo apt-get install libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
After that go the directory where your python file is then reconfigure and reinstall python3.6 .
cd /opt/Python3.6/
./configure
make
sudo make install
NOTE
If you installed Python3.6 via ppa, then reinstall it again;
sudo apt-get install python3.6
Now you should be able to use pip3.6
I ran into the same error when building Python 3.6.1 from source under CentOS 7.
For CentOS7, I had to first:
sudo yum install openssl-dev
Then:
./configure --enable-optimizations
make altinstall
Now pip3.6 works :-)
A complete script can be found HERE
Install Prerequisites
For RHEL/CentOS
sudo yum -y install gcc gcc-c++ zlib zlib-devel libffi-devel openssl-devel wget
For Ubuntu/Debian
sudo apt-get -y install build-essential python-dev python-setuptools python-pip
python-smbus libncursesw5-dev libgdbm-dev libc6-dev zlib1g-dev libsqlite3-dev
tk-dev libssl-dev openssl libffi-dev wget
Download Python
Modify for the version of python you want
Python Versions
cd /var/tmp
sudo wget https://www.python.org/ftp/python/3.x.x/Python-x.x.x.tgz
sudo tar xf Python-3.*.tgz
cd Python-3*
Configure/Make/Install
sudo ./configure --enable-optimizations --enable-shared --prefix=/usr/local
sudo make && make altinstall
Cleanup Shared Library & Add to Path
Stripping the shared library of debugging symbols can speed up execution when running parallel scripts.
sudo make && make altinstall
sudo strip /usr/local/lib/libpython3.7m.so.1.0
sudo echo 'export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib' >> /etc/profile.d/python.sh
sudo echo 'export PATH=${PATH}:~/usr/local/bin/' >> /etc/profile.d/python.sh
sudo echo '/usr/local/lib' >> /etc/ld.so.conf
sudo ldconfig
Reference
Gist
Stack Overflow
Stack Overflow
Daniel Erikson
Unix StackExchange
TLDP
The steps I've taken are below. During the ./configure step, unless I allow the system default for the cuDNN version (v5) instead of specifying v5.1.5 (as I want), I get an error saying that the environment version (v5) for cuDNN doesn't match the target version (5.1.5). Since I have downloaded the cuDNN v5.1.5 file for CUDA 8.0 from Nvidia I'm baffled.
What's more, after digging through https://github.com/tensorflow/tensorflow/blob/master/third_party/gpus/cuda_configure.bzl around line 240, I'm seeing that the version the bazel configure script checks against seems to only take into account the Major numeral of the cuDNN version (my Major, Minor and Patch numerals within cudnn.h are 5, 1, 5 respectively). So, I'm uncertain as to how it would ever allow version 5.1.5 to be installed. Maybe I'm looking at it wrong?
In any case, does anyone have a method of installing cuDNN 5.1.5 with CUDA 8.0 and Tensorflow 0.12 on a g2.2xlarge instance?
Thanks!
Steps
(Note: these work, but it installs cuDNN 5.0, not 5.1.5 as desired)
Provisioning
follow provisioning steps from: https://medium.com/#giltamari/tensorflow-getting-started-gpu-installation-on-ec2-9b9915d95d6f#.2hv67eeek
(i.e. up to but not including: sudo apt-get update && sudo apt-get -y upgrade)
Installing dependencies and tools
familiarize self with:
http://expressionflow.com/2016/10/09/installing-tensorflow-on-an-aws-ec2-p2-gpu-instance/
http://ramhiser.com/2016/01/05/installing-tensorflow-on-an-aws-ec2-instance-with-gpu-support/
https://alliseesolutions.wordpress.com/2016/09/08/install-gpu-tensorflow-from-sources-w-ubuntu-16-04-and-cuda-8-0-rc/
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install -y build-essential git python-pip libfreetype6-dev libxft-dev libncurses-dev libopenblas-dev gfortran python-matplotlib libblas-dev liblapack-dev libatlas-base-dev python-dev python-pydot linux-headers-generic linux-image-extra-virtual unzip python-numpy swig python-pandas python-sklearn unzip wget pkg-config zip g++ zlib1g-dev libcurl3-dev
sudo pip install -U pip
Installing Cuda 8
wget https://developer.nvidia.com/compute/cuda/8.0/prod/local_installers/cuda-repo-ubuntu1604-8-0-local_8.0.44-1_amd64-deb
sudo dpkg -i cuda-repo-ubuntu1604-8-0-local_8.0.44-1_amd64-deb
rm cuda-repo-ubuntu1604-8-0-local_8.0.44-1_amd64-deb
sudo apt-get update
sudo apt-get install -y cuda
Installing cuDNN
We want to download and install latest version of cuDNN. Downloading cuDNN requires logging into NVIDIA developer site, so we can’t use wget to fetch the files. Download the following files from NVIDIA and upload them to your AWS instance.
download cuDNN 5.1 for CUDA 8.0 on linux
scp -i ssh-key.pem path/to/downloaded/cudnn ubuntu#ec2{instance}.us-west-1.compute.amazonaws.com:~/
sudo tar -xzvf cudnn-8.0-linux-x64-v5.1.tgz
sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
Configure the Environment
put following at end of ~/.bashrc:
export CUDA_HOME=/usr/local/cuda
export CUDA_ROOT=/usr/local/cuda
export PATH=$PATH:$CUDA_ROOT/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDA_ROOT/lib64:$CUDA_ROOT/extras/CUPTI/lib64
source ~/.bashrc
sudo reboot
Installing Bazel
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update
echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf- set-selections
echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf- set-selections
sudo apt-get install -y oracle-java8-installer
sudo apt-get install pkg-config zip g++ zlib1g-dev
scp https://github.com/bazelbuild/bazel/releases/download/0.3.2/bazel-0.3.2-installer-linux-x86_64.sh from local machine to ec2 instance
chmod +x bazel-0.1.4-installer-linux-x86_64.sh
./bazel-0.1.4-installer-linux-x86_64.sh --user
rm bazel-0.1.4-installer-linux-x86_64.sh
bazel version
Building and Installing Tensorflow
git clone --recurse-submodules https://github.com/tensorflow/tensorflow
cd tensorflow
TF_UNOFFICIAL_SETTING=1 ./configure
hit enter/default for all except:
CUDA version = 8.0, CUDA dependency = 3.0 (k520 gpu)
bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
sudo pip install --upgrade /tmp/tensorflow_pkg/tensorflow-0.12.0rc1-cp27-cp27mu-linux_x86_64.whl