Permission denied when trying to install inside a virtualenv using ansible - ansible

Question

I have this play in my ansible playbook
- name: Setup virtualenv
pip:
requirements="/vagrant/webapp/requirements/{{ requirements_filename }}"
virtualenv="$HOME/.envs/{{ project_name }}"
However, I am receiving a Permission denied error. It seems like ansible is trying to install the package into the system path and not into the virtualenv.
Here's the full traceback:
TASK: [deploy | Setup virtualenv] *********************************************
failed: [default] => {"cmd": "/usr/bin/pip install -r /vagrant/webapp/requirements/dev.txt", "failed": true}
msg: stdout: New python executable in /home/vagrant/.envs/venv/bin/python
Installing setuptools, pip...done.
Downloading/unpacking django>=1.8 (from -r /vagrant/webapp/requirements/requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): wsgiref>=0.1.2 in /usr/lib/python2.7 (from -r /vagrant/webapp/requirements/requirements.txt (line 4))
Downloading/unpacking django-allauth>=0.19.1 (from -r /vagrant/webapp/requirements/requirements.txt (line 5))
Running setup.py (path:/tmp/pip_build_vagrant/django-allauth/setup.py) egg_info for package django-allauth
Downloading/unpacking psycopg2>=2.6 (from -r /vagrant/webapp/requirements/requirements.txt (line 6))
Running setup.py (path:/tmp/pip_build_vagrant/psycopg2/setup.py) egg_info for package psycopg2
Downloading/unpacking django-currencies>=0.3.3 (from -r /vagrant/webapp/requirements/requirements.txt (line 7))
Downloading django-currencies-0.3.3.tar.gz
Running setup.py (path:/tmp/pip_build_vagrant/django-currencies/setup.py) egg_info for package django-currencies
Downloading/unpacking django-countries>=3.2 (from -r /vagrant/webapp/requirements/requirements.txt (line 8))
Running setup.py (path:/tmp/pip_build_vagrant/django-countries/setup.py) egg_info for package django-countries
Downloading/unpacking stripe>=1.22.1 (from -r /vagrant/webapp/requirements/requirements.txt (line 9))
Running setup.py (path:/tmp/pip_build_vagrant/stripe/setup.py) egg_info for package stripe
Downloading/unpacking pillow>=2.8.0 (from -r /vagrant/webapp/requirements/requirements.txt (line 10))
Running setup.py (path:/tmp/pip_build_vagrant/pillow/setup.py) egg_info for package pillow
Single threaded build, not installing mp_compile: 1 processes
warning: no files found matching 'LICENSE' under directory 'docs'
Downloading/unpacking django-compressor>=1.4 (from -r /vagrant/webapp/requirements/requirements.txt (line 11))
Downloading/unpacking djangorestframework>=3.1.1 (from -r /vagrant/webapp/requirements/requirements.txt (line 12))
Downloading/unpacking django-filter>=0.9.2 (from -r /vagrant/webapp/requirements/requirements.txt (line 13))
Downloading django_filter-0.9.2-py2.py3-none-any.whl
Obtaining formtools from git+https://github.com/django/django-formtools.git#egg=formtools (from -r /vagrant/webapp/requirements/requirements.txt (line 14))
Cloning https://github.com/django/django-formtools.git to ./src/formtools
Running setup.py (path:/tmp/src/formtools/setup.py) egg_info for package formtools
Installing extra requirements: 'egg'
Downloading/unpacking django-widget-tweaks>=1.3 (from -r /vagrant/webapp/requirements/requirements.txt (line 15))
Downloading django-widget-tweaks-1.3.tar.gz
Running setup.py (path:/tmp/pip_build_vagrant/django-widget-tweaks/setup.py) egg_info for package django-widget-tweaks
Downloading/unpacking django-braces>=1.4.0 (from -r /vagrant/webapp/requirements/requirements.txt (line 16))
Downloading django_braces-1.4.0-py2.py3-none-any.whl
Downloading/unpacking django-debug-toolbar>=1.3.0 (from -r /vagrant/webapp/requirements/dev.txt (line 4))
Downloading/unpacking python-openid>=2.2.5 (from django-allauth>=0.19.1->-r /vagrant/webapp/requirements/requirements.txt (line 5))
Running setup.py (path:/tmp/pip_build_vagrant/python-openid/setup.py) egg_info for package python-openid
warning: no files found matching 'CHANGELOG'
warning: no files found matching '*.css' under directory 'doc'
warning: no files found matching '*.html' under directory 'doc'
Downloading/unpacking requests-oauthlib>=0.3.0 (from django-allauth>=0.19.1->-r /vagrant/webapp/requirements/requirements.txt (line 5))
Downloading requests_oauthlib-0.4.2-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): requests>=1.0.3 in /usr/lib/python2.7/dist-packages (from django-allauth>=0.19.1->-r /vagrant/webapp/requirements/requirements.txt (line 5))
Downloading/unpacking django-appconf>=0.4 (from django-compressor>=1.4->-r /vagrant/webapp/requirements/requirements.txt (line 11))
Downloading django_appconf-1.0.1-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): six in /usr/local/lib/python2.7/dist-packages (from django-braces>=1.4.0->-r /vagrant/webapp/requirements/requirements.txt (line 16))
Downloading/unpacking sqlparse (from django-debug-toolbar>=1.3.0->-r /vagrant/webapp/requirements/dev.txt (line 4))
Running setup.py (path:/tmp/pip_build_vagrant/sqlparse/setup.py) egg_info for package sqlparse
Downloading/unpacking oauthlib>=0.6.2 (from requests-oauthlib>=0.3.0->django-allauth>=0.19.1->-r /vagrant/webapp/requirements/requirements.txt (line 5))
Running setup.py (path:/tmp/pip_build_vagrant/oauthlib/setup.py) egg_info for package oauthlib
Installing collected packages: django, django-allauth, psycopg2, django-currencies, django-countries, stripe, pillow, django-compressor, djangorestframework, django-filter, formtools, django-widget-tweaks, django-braces, django-debug-toolbar, python-openid, requests-oauthlib, django-appconf, sqlparse, oauthlib
Cleaning up...
Exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
status = self.run(options, args)
File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 283, in run
requirement_set.install(install_options, global_options, root=options.root_path)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1435, in install
requirement.install(install_options, global_options, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 671, in install
self.move_wheel_files(self.source_dir, root=root)
File "/usr/lib/python2.7/dist-packages/pip/req.py", line 901, in move_wheel_files
pycompile=self.pycompile,
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 206, in move_wheel_files
clobber(source, lib_dir, True)
File "/usr/lib/python2.7/dist-packages/pip/wheel.py", line 193, in clobber
os.makedirs(destsubdir)
File "/usr/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/Django-1.8.dist-info'
Storing debug log for failure in /home/vagrant/.pip/pip.log
FATAL: all hosts have already failed -- aborting

Answer 1

I solved the problem by setting the path explicitly and not using the $HOME variable.
- name: Setup virtualenv
pip:
virtualenv="/home/vagrant/.envs/{{ project_name }}"
requirements="/vagrant/webapp/requirements/{{ requirements_filename }}"

Answer 2

Try specifying a pip executable path as well:
executable="$HOME/.envs/{{ project_name }}/bin/pip"
Didn't search for an exact reason TBH, but had to do the same to get stuff installed into virtual environments. Seemed like it tried to install into a virtual environment using global pip. Maybe that's possible to do under some circumstances, wasn't my case.

Answer 3

It seems the pip module is using the system Python so it will try and install the package on the expected system path to site-packages and since you're not running this as root or using sudo it can't write to '/usr/local/lib/python2.7/...'
I'm assuming you want to install the pip package under the virtualenv, correct?
If so beware that Ansible modules always use, by default, the system default Python interpreter unless you set ansible_python_interpreter.
An example of how to do this can be found on Best way to always run ansible inside a virtualenv on remote machines?
A link to a short discussion about the subject and why Ansible people don't want to change #!/usr/bin/python for the more virtualenv-friendly #!/usr/bin/env python can be found here.
As a note you'll have the same problem if you're running Ansible from a local virtualenv and try and use say ec2 module without having boto installed at the system-level (but installed at the virtualenv level).

Answer 4

Looks like it is using /usr/bin/pip, not pip from virtualenv. Not sure why this happens - a bug in Ansible? Can you manually check your virtualenv has pip command? One thing I can think out from my head pip is created for root (sudo) and you try to run the command as a normal user and it doesn't have read access to virtualenv'ed pip.
You can probably use pip manually like this for workaround.
- name: Create virtualenv for app using chosen Python
shell: virtualenv --python=python3.4 {{deploy_location}}/venv
sudo: yes
- name: Install Python packages
sudo: yes
command: {{deploy_location}}/venv/bin/pip -r {{deploy_location}}/requirements.txt