Encoding of requirement file using pip - macos

Question

On OSX (El Capitan, 10.11.6), using virtualenv (15.1.0), I am getting an error when installing requirements from a text file with pip (9.0.1):
virtualenv env
source env/bin/activate
pip install -r requirements.txt
but not when looping over each requirement manually:
for r in $(cat requirements.txt); do pip install "$r"; done
This makes me think there might be an issue with the default encoding assumed by pip when reading the requirements file. Is there a way (environment variable, I presume) to set the default encoding of requirement files?
The error I get is:
Exception:
Traceback (most recent call last):
File "/path/to/env/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/path/to/env/lib/python2.7/site-packages/pip/commands/install.py", line 312, in run
wheel_cache
File "/path/to/env/lib/python2.7/site-packages/pip/basecommand.py", line 295, in populate_requirement_set
wheel_cache=wheel_cache):
File "/path/to/env/lib/python2.7/site-packages/pip/req/req_file.py", line 84, in parse_requirements
filename, comes_from=comes_from, session=session
File "/path/to/env/lib/python2.7/site-packages/pip/download.py", line 422, in get_file_content
content = auto_decode(f.read())
File "/path/to/env/lib/python2.7/site-packages/pip/utils/encoding.py", line 31, in auto_decode
return data.decode(locale.getpreferredencoding(False))
LookupError: unknown encoding:
The following test code:
#!/usr/bin/env python
import sys
import locale
print sys.stdin.encoding
print locale.getpreferredencoding()
print locale.getpreferredencoding(False)
print sys.getdefaultencoding()
print sys.getfilesystemencoding()
returns:
None
US-ASCII
ascii
utf-8
From the command-line:
$ locale
LANG="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_CTYPE="utf-8"
LC_MESSAGES="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_ALL=

Answer 1

According to your test the locale.getpreferredencoding() is absent — it should look like this:
UTF-8
UTF-8
US-ASCII
ascii
utf-8
Doing locale mine looks almost like yours (other than US vs GB and LC_CTYPE). It does seem peculiar that LC_CTYPE seems different, and maybe worth looking into as to why.
$ locale
LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL=
There are a couple of things you can try to correct the issue, the first being your shebang:
#!/usr/bin/python
Try changing it to this perhaps, as #!/usr/bin/env python might not set the locale properly. If that doesn't work you can always try forcing the encoding within your script:
import locale
loc = locale.getlocale()
locale.setlocale(locale.LC_ALL, '') # use user's preferred locale
locale.setlocale(locale.LC_ALL, 'C') # use default (C) locale
locale.setlocale(locale.LC_ALL, loc) # restore saved locale
# OR
locale.getpreferredencoding(do_setlocale=True) or "utf-8"
# OR
if locale.getpreferredencoding() == '':
locale.setlocale(locale.LC_ALL,'UTF-8')
Ultimately you'll want to figure out why locale.getpreferredencoding() is coming up empty for the first couple tests, and why you have a mismatched LC_CTYPE set in your locale.