I'm trying to write a short python script which fetches data from Google Places API and exports it as a .csv file.
Unfortunately I'm stuck right at the beginning.
I want to use requests package but my machine cannot find it.
I've installed Python 3.5 on my Mac and when I use pip to install requests it says:
Requirement already satisfied: requests in /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages
But when I run the program I get an import error which says:
ImportError: No module named requests
Do you have any ideas how i can fix this issue?
Thanks in advance.
Much love.
Use python -m pip install requests instead. Your pip installation was probably for a different version.
I want to run a script that compiles my sass in a "build" container.
Since this container will repeatedly be restarted I need a robust and quick way to install or use sass. (including ruby and dependencies)
Is there a simple way for sass to be available in a container for scss compiling?
There is a solution using a ruby container but this is not possible in my case since i already need a specific container image for the build itself.
Using another container and named volumes is also not possible in my case.
You can make a Docker image that inherits the specific Ruby image you need, add your changes to it, and use it instead.
For example, if you currently use debian:jessie, then you can create a new Dockerfile with:
FROM debian:jessie
# Now you install all the dependencies including Ruby... etc
RUN gem install sass
Then you can build and image from that file, and publish that image to a docker registry, and use it instead of the one you are currently using.
Take a look at the source code of some Ruby docker image to see how you can install Ruby and dependencies. Ofcourse, you are free to install it however you want (natively, using rbenv or using rvm... etc).
I am new to docker but managed to build myself some dev-environment images ( which is awesome! ). But i wasn't quite satisfied with the filesize of the resulting image, so i tried to migrate the image from node-argon image ( based on debian-wheezy ) to alpine image. Problem is that the installation of vagrant.deb package isn't working correctly. I installed the alpine dpkg package, but get these errors:
dpkg: error: failed to open package info file '/var/lib/dpkg/status' for reading: No such file or directory
I found some threats about this topic, but were not related to alpine installation and wouldn't fix my issues. Relevant docker lines are:
FROM gliderlabs/alpine:3.3
RUN apk add --update dpkg
ENV VAGRANT_VERSION 1.8.1
ADD https://releases.hashicorp.com/vagrant/${VAGRANT_VERSION}/vagrant_${VAGRANT_VERSION}_x86_64.deb .
RUN dpkg -i vagrant_${VAGRANT_VERSION}_x86_64.deb
RUN rm vagrant_*.deb
Someone else got this working - my guess was that there maybe some build/install dependencies missing, but couldn't get it any further. Any advice would be much appriciated.
All the best, florian
In comments below the question we found out, that it is indeed possible to install the vagrant.deb using dpkg on Alpine Linux. However, it was really hackish(!) and at the end the advantage in size of the Alpine Linux didn't matter since the resulting image had a size of 2.5G =).
That's why my answer is (I said that before): Make your life easy and use a Debian or Ubuntu image if you want to install deb packages :)
For all potential Alpine-container users, please think twice if you really need a base image that is a couple of MB smaller than common GNU/Linux base images. You'll pay the price for that few MBs with having a poor shell environment (Welcome to the 80s!) and a significantly smaller amount of available packages.
If you want a minimal container then don't put a distribution into a container, put a process into a container!
I have a flask app where I'm trying to automate deployment to EC2.
Not a big deal, but is there a setting in either Fabric or Distribute that reads the requirements.txt file directly for the setup.py, so I don't have to spell everything out in the setup(install_requires=[]) list, rather than writing a file reader for my requirements.txt? If not, do people have recommendations or suggestions on auto-deployment and with pip?
I'm reviewing from here and here.
Not a big deal, but is there a setting in either Fabric or Distribute
that reads the requirements.txt file directly for the setup.py, so I
don't have to spell everything out in the setup(install_requires=[])
list, rather than writing a file reader for my requirements.txt?
You might still want to checkout frb's answer to the duplicate question How can I reference requirements.txt for the install_requires kwarg in setuptools.setup?, which provides a straight forward two line solution for writing a file reader.
If you really want to avoid this, you could alternatively add the common pip install -r requirements.txtto your fabfile.py, e.g.:
# ...
# create a place where we can unzip the tarball, then enter
# that directory and unzip it
run('mkdir /tmp/yourapplication')
with cd('/tmp/yourapplication'):
run('tar xzf /tmp/yourapplication.tar.gz')
# now install the requirements with our virtual environment's
# pip installer
run('/var/www/yourapplication/env/scripts/pip install -r requirements.txt')
# now setup the package with our virtual environment's
# python interpreter
run('/var/www/yourapplication/env/bin/python setup.py install')
I got Numpy and Matplotlib running on Heroku, and I'm trying to install Scipy as well. However, Scipy requires BLAS[1] to install, which is not presented on the Heroku platform. After contacting Heroku support, they suggested me to build BLAS as a static library to deploy, and setup the necessary environment variables.
So, I compiled libblas.a on a 64bit Linux box, and set the following variables as described in [2] :
$ heroku config
BLAS => .heroku/vendor/lib/libfblas.a
LD_LIBRARY_PATH => .heroku/vendor/lib
LIBRARY_PATH => .heroku/vendor/lib
PATH => bin:/usr/local/bin:/usr/bin:/bin
PYTHONUNBUFFERED => true
After adding scipy==0.10.1 in my requirements.txt, the push still fails.
File "scipy/integrate/setup.py", line 10, in configuration
blas_opt = get_info('blas_opt',notfound_action=2)
File "/tmp/build_h5l5y31i49e8/lib/python2.7/site-packages/numpy/distutils/system_info.py", line 311, in get_info
return cl().get_info(notfound_action)
File "/tmp/build_h5l5y31i49e8/lib/python2.7/site-packages/numpy/distutils/system_info.py", line 462, in get_info
raise self.notfounderror(self.notfounderror.__doc__)
numpy.distutils.system_info.BlasNotFoundError:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
It seem that pip is not aware of the BLAS environment variable, so I check the environment using heroku run python:
(venv)bash-3.2$ heroku run python
Running python attached to terminal... import up, run.1
Python 2.7.2 (default, Oct 31 2011, 16:22:04)
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.system('bash')
~ $ echo $BLAS
.heroku/vendor/lib/libfblas.a
~ $ ls .heroku/vendor/lib/libfblas.a
.heroku/vendor/lib/libfblas.a
~ $
And it seems fine. Now I have no idea how to solve this.
[1] http://www.netlib.org/blas/
[2] http://www.scipy.org/Installing_SciPy/Linux
I managed to get this working on the cedar stack by building numpy and scipy offline as bdists and then modifying the heroku python buildpack to unzip these onto the dyno's vendor/venv areas directly. You can also use the buildpack to set environment variables that persist.
Heroku haven't officially published buildpacks yet - search for 'heroku buildpacks' for more thirdparty/heroku ones and information.
My fork of the python build pack is here:
https://wyn#github.com/wyn/heroku-buildpack-python.git
The changes are in the bin/compile where I source two new steps, a scipy/numpy step and an openopt step. The scripts for the two steps are in bin/steps/npscipy and bin/steps/openopt. I also added some variables to bin/release. Note that I am assuming installation through a setup.py file rather than the requirements.txt approach (c.f. https://devcenter.heroku.com/articles/python-pip#traditional_distributions).
I also download the blas/lapack/atlas/gfortran binaries that were used to build numpy/scipy onto the dyno as there are c extensions that need to link through to them. The reason for building everything offline and downloading is that pip-installing numpy/scipy requires you to have a fortran compiler + associated dev environment and this made my slugs too big.
It seems to work, the slug size is now 35mb and scaling seems fast too. All but one of the numpy tests pass and all of the scipy tests pass.
This is still work in progress for me but I thought I'd share.
In case someone else comes to this like I did...
The excellent answer on this question from #coshx sadly no longer works (at least I could not get it to work). What I did, however was the following:
I forked npsicpy-binaries repository from #coshx and changed all the tar files such that they do not have venv as the root folder inside (my fork is here)
I forked the npsp-helloworld repository from #coshx and made it use a requirements.txt file instead of the setup.py (my fork is here - this means that you can use the whole pip approach).
I forked the heroku-buildpack-python repository from Heroku, took the npscipy file from #coshx and changed it to work with this latest version of the build pack (my fork is here - you can see that there is no venv set up, for example).
After doing those three things I have the npsp-helloworld application working perfectly. You just need to make sure you set the buildpack appropriately when creating the application and you are good to go:
$ heroku create --stack=cedar --buildpack=https://github.com/kmp1/heroku-buildpack-python.git
NOTE: I haven't worked out how to make my own binaries yet, so the three libraries (scipy, numpy and scikit-learn) are not the latest versions so make sure when you install it you do (if anyone can build these I would gladly accept a pull request for them):
pip install scipy==0.11.0
pip install numpy==1.7.0
pip install scikit-learn==0.13.1
By the way - I am really sorry if I have not done things in the correct way, etiquette-wise. I'm still learning git and this whole open source thing.
Another good option is the conda buildpack, which allows you to add any of the free Linux64 packages available through Anaconda/Miniconda to a Heroku app. Some of the most popular packages include numpy, scipy, scikit-learn, statsmodels, pandas, and cvxopt. While the buildpack makes it fairly simple to add packages to an app, the downsides are that the buildback takes up a lot of space and that you have to wait on Anaconda to update the libraries in the repository.
If you are starting a new Python app on Heroku, you can add the conda buildpack using the command:
$ heroku create YOUR_APP_NAME --buildpack https://github.com/kennethreitz/conda-buildpack.git
If you have already setup a Python app on Heroku, you can add the conda buildpack to the existing app using the command:
$ heroku config:add BUILDPACK_URL=https://github.com/kennethreitz/conda-buildpack.git
Or, if you need to specify the app by name:
$ heroku config:add BUILDPACK_URL=https://github.com/kennethreitz/conda-buildpack.git --app YOUR_APP_NAME
To use the buildpack, you will need to include two text files in the app directory, requirements.txt and conda-requirements.txt. Just as with the standard Python buildpack, the requirements.txt file lists packages that should be installed using pip. Packages that should be installed using conda are listed in the conda-requirements.txt file. Some of the most useful scientific packages include numpy, scipy, scikit-learn, statsmodels, pandas, and cvxopt. The full list of available conda packages can be found at repo.continuum.io.
For example:
$ cat requirements.txt
gunicorn==0.14.2
requests==0.11.1
$ cat conda-requirements.txt
scipy
numpy
cvxopt
That’s it! You can now add Anaconda packages to a Python app on Heroku.
The slug compiler is not aware of your environment variables, which is why it fails during push, and not once running.
The only real option you have is to look at the user_env_compile addon that's currently in labs beta.
http://devcenter.heroku.com/articles/labs-user-env-compile
For those who wish to use Python 3.4 in production, I've built numpy 1.8.1, scipy 0.14.0, and scikit-learn 0.15-git (0.14 doesn't work with the others for some reason) as binaries on Ubuntu 10.04 LTS 64-bit, which works on the Heroku cedar stack. Here is the git repo.
My heroku buildpack is quite similar to that of kmp. Note that the bin/steps/npscipy file links to my repository of binaries above.
I'm putting this here in case someone stumbled on this like I do. Since I am using python 3.4, thenovices buildpack doesn't work out for me.
I tried to look at conda buildpack, but it seems like a little bit of an overkill for my application which only requires scipy and numpy. What finally works for me is this excellent guide, using a multi buildpacks approach. The scipy installation was indeed a bit long though. Hope this helps! :)