I am trying to get to grips with Python on my Mac running OS X 10.15.7 Catalina.
My goal is to be able to run a program requiring Python 3.9 and associated libraries, in its own virtual environment, using venv.
My quest has led me over many rocky paths and blind alleys - including via Anaconda and Brew.
I am running zsh - specifically oh-my-zsh ys version. If that's important. Most online forums and tutorials assume bash. I would prefer to do all in zsh to avoid having to edit both zsh and bash profiles.
Current Shell Outputs
$ python -V
Python 2.7.16
$ which python
/usr/bin/python
$ python3 -V
Python 3.8.2
$ which python3.8
python3.8 not found
$ which python3.9
python3.9 not found
$ python3 -m -V
/Library/Developer/CommandLineTools/usr/bin/python3: No module named -V
Pausing there ..
Py 2.7 is the OS X system Python and I don't want to do anything with it.
It is odd that python3 -m returns a reference to the CLI tools. See below. The only location for py3.8 is in the CL tools.
What pythons do I have (output below is edited so as to show only main directories and Files)?
$ locate python
/Applications/LibreOffice.app/Contents/Frameworks/LibreOfficePython.framework/Versions/3.7/bin/python3
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/bin/python3.8
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python
/Library/Developer/CommandLineTools/usr/bin/python3.8
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3-config
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9-config
/Library/Frameworks/Python.framework/Versions/3.9/share/man/man1/python3.1
/Library/Frameworks/Python.framework/Versions/3.9/share/man/man1/python3.9.1
/Library/Frameworks/SAVI.framework/Versions/A/Frameworks/Python.framework/Versions/3.6/lib/python3.6
/usr/bin/python
/usr/bin/python-config
/usr/bin/python2
/usr/bin/python2.7
/usr/bin/python2.7-config
/usr/bin/python3
/usr/bin/pythonw
/usr/bin/pythonw2.7
/usr/lib/libpython.dylib
/usr/lib/libpython2.7.dylib
/usr/lib/python2.7
/usr/local/bin/python3
/usr/local/bin/python3-config
/usr/local/bin/python3.9
/usr/local/bin/python3.9-config
/usr/share/file/magic/python
/usr/share/man/man1/python.1
/usr/share/man/man1/python2.7.1
/usr/share/man/man1/pythonw.1
/usr/share/man/man1/pythonw2.7.1
/usr/share/vim/vim81/autoload/python3complete.vim
/usr/share/vim/vim81/autoload/pythoncomplete.vim
/usr/share/vim/vim81/ftplugin/python.vim
/usr/share/vim/vim81/indent/python.vim
/usr/share/vim/vim81/syntax/python.vim
/usr/share/zsh/5.7.1/functions/_bpython
/usr/share/zsh/5.7.1/functions/_python
/usr/share/zsh/5.7.1/functions/_python_modules
WHAT ABOUT PIP?
$ pip3
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
$ pip3 install --upgrade pip
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in ./Library/Python/3.8/lib/python/site-packages (20.3.3)
Note two points
Although the wrapper warning comes up, PIP 20.3.3 is latest version which I have upgraded. Probably the reason for the warning is in the next lines, which suggest that PIP3 is being invoked from the CL Tools.
2; The ‘normal site-packages is not writeable’ error is supposed to be cured by specifying the Path. I have tried already to specify the correct path - see below - but this has not cured the problem and is another major reason for asking the question at the end of this post.
$ python -m pip
/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named pip
$ python3 -m pip
Usage:
/Library/Developer/CommandLineTools/usr/bin/python3 -m pip <command> [options]
$ which pip3
/usr/bin/pip3
$ locate pip3
/Applications/LibreOffice.app/Contents/Frameworks/LibreOfficePython.framework/Versions/3.7/bin/pip3
/Applications/LibreOffice.app/Contents/Frameworks/LibreOfficePython.framework/Versions/3.7/bin/pip3.7
/Library/Developer/CommandLineTools/usr/bin/pip3
/Library/Developer/CommandLineTools/usr/bin/pip3.8
/Library/Frameworks/Python.framework/Versions/3.9/bin/pip3
/Library/Frameworks/Python.framework/Versions/3.9/bin/pip3.9
/usr/bin/pip3
/usr/local/bin/pip3
/usr/local/bin/pip3.9
SYSTEM FILES
~/.zprofile contents - edited a few times to remove references to virfualenv and penv, both originally installed with brew and now removed. I have run brew uninstall and brew remove to try to clean everything up but may not have succeeded.
Start
# THIS FILE COMMENTED OUT RH 21DEC20
## Setting PATH for Python 3.9
## The original version is saved in .zprofile.pysave. **Note in this post only - cannot find this file**
PATH="/Library/Frameworks/Python.framework/Versions/3.9/bin:${PATH}"
export PATH **Note in this post only - this does not seem to have worked - see above Python3 -V**
# export WORKON_HOME=~/.virtualenvs
# mkdir -p $WORKON_HOME
# . ~/.pyenv/versions/3.9.0/bin/virtualenvwrapper.sh
# eval "$(pyenv init -)"
# # AND THESE LINES ADDED FOLLOWING STACKOVERFLOW Q 33216679 **Note in this post only - commented out as shell threw errors .’cannot find virtualenvwrapper.sh**
# export WORKON_HOME=$HOME/.Envs
# # export VIRTUALENVWRAPPER_PYTHON=$/usr/bin/python3
# export VIRTUALENVWRAPPER_PYTHON=$/usr/local/bin/python3
#source $HOME/.local/bin/virtualenvwrapper.sh
END
Finally a word about Brew. Many tutorials advised the use of Brew to install Python, pyenv, and libraries etc. I did that and successfully created a test environment, but could not the utilise it. I have then removed as much as possible in order to get back to a ‘clean’ state to work with.
Current brew status is
$ brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!
Warning: Unbrewed dylibs were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.
Unexpected dylibs:
/usr/local/lib/libdvdcss.2.dylib
/usr/local/lib/libfuse.0.dylib
/usr/local/lib/libulockmgr.0.dylib
Warning: Unbrewed header files were found in /usr/local/include.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.
Unexpected header files:
/usr/local/include/ulockmgr.h
Warning: Unbrewed .la files were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.
Unexpected .la files:
/usr/local/lib/libulockmgr.la
Warning: Unbrewed static libraries were found in /usr/local/lib.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.
Unexpected static libraries:
/usr/local/lib/libfuse.a
/usr/local/lib/libulockmgr.a
SUMMARY
My first goal is to set Py3.9.x as default for new projects. I think also this would require/imply that pip and other commands run from v3.9 and not any other version or location. Must admit to being hazy about that.
Then I want to be able to create a virtual environment into which I can put the program I want to run and its associated libraries and packages. Doing this implies using terminal, but I need to know which directory I need to be in to run which commands.
At this point I have also been running into confusion with tutorials on how to create a virtual environment. From what I have read I think I should use venv over pyenv. What is not clear is whether I run the venv command (s) from root, or whether I have to create and cd to a different location, and if so what.
Any help in
1 Changing the default Python to 3.9
2 Creating a venv to use v3.9.
3 What commands to run where.
Will be greatly appreciated. Please keep it simple and don't assume any prior knowledge!
This is what I would do :
# For 1 Changing the default python to 3.9
cd /usr/local/bin
sudo ln -fs /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 python3
sudo ln -fs python3 python
# And make sure /usr/local/bin is before (/usr/bin and /Library/Developer/CommandLineTools/usr/bin) in your $PATH
# For 2 Creating a venv to use v3.9
/usr/local/bin/python3.9 -m venv /tmp/python3.9-venv
Related
Most of you may know OpenAI playground, so I built an function generator app and followed all the instructions but I can't launch it via venv-python. I can actually install the requirements using pip while venv is not active and can launch the site with flask, but when I try to install requirements after activating venv, I get this error;
User#lalec ~
$ cd openai-quickstart-python
User#lalec ~/openai-quickstart-python (master)
$ . venv/Scripts/activate
(venv)
User#lalec ~/openai-quickstart-python (master)
$ pip install -r requirements.txt
Fatal error in launcher: Unable to create process using '"C:\Users\celal\openai-quickstart-python\venv\Scripts\python.exe" "C:\Users\User\openai-quickstart-python\venv\Scripts\pip.exe" install -r requirements.txt': The system cannot find the file specified.
How do I fix this? I added every possible script locations into PATH I thought it would help but no result.
I just realized two directories in the error doesn't match, and "C:\Users\celal\openai-quickstart-python\venv\Scripts\python.exe" actually does not even exist. Maybe that's the problem... How can I change this ??
Sorry if I'm asking dumb questions I'm new to all this... Also, I don't get why I need venv activated while I can just launch it by accessing the directory and type flask run in cmd, would appreciate any answers.
ss of the project's directory
ss of venv/Scripts/
ss of bash screen with errors
The error message suggests that the requirements.txt file can't be read. Check that you definitely can read it and it's in the expected location.
The perferred command for running pip is:
python3 -m pip install -r requirements.txt
This makes sure that you know which python3 binary you're running pip with - some systems have more than one python3 binary in various locations.
You could achieve similar without activating the virtual environment:
./venv/bin/python3 -m pip install -r requirements.txt
I'm trying to install lektor into a new virtualenv using pyenv virtualenv. I have no problem making a new virtualenv with Python 3.7.0, but when I try to pip install lektor, I get a build failure with watchdog and the following error message:
pyenv: clang: command not found
The `clang' command exists in these Python versions:
anaconda3-5.0.0
error: command 'clang' failed with exit status 127
I understand that the anaconda3 version has clang, but I also have a version installed in /usr/bin. The output of which -a clang is:
$PYENV_ROOT/shims/clang
/usr/bin/clang
Can I manually set the shim to point to usr/bin? Or is there a different solution to this issue?
This is a rotten solution, but it does work. By taking the shim for clang and temporarily moving it (e.g., mv $PYENV_ROOT/shims/clang ~/Desktop), watchdog can be installed, then the clang shim can be moved back to the original location. A better option would be for pyenv to default to the system versions of programs if a shim doesn't point anywhere in a given environment…
Alternative solution
The pyenv plugin pyenv-which-ext takes care of this class of issues by checking for the existence of utilities in the normal $PATH after seeing that they aren't present in the current environment. It's a quick install via Homebrew, took care of all my issues.
I am trying to install ROOT (cern.root.ch). When I run ./configure , I get a message that libX11 is missing and must be installed.
I did some research and found that I need to install
) XQuartz (I already have the latest version.)
) Command line tools in Xcode.
I tried installing Command Line Tools from apple's developer website. The installation goes through smoothly but how do I know whether it has been installed? I still get libX11 missing error with root's configure command.
I also tried xcode-select --install and it once went through smoothly and then later again gives error saying this package is no longer maintained - or something of that sort.
I understand I may have multiple installations... But I am still facing the problem of not having libX11 and not being able to install ROOT.
Thanks,
Hershal.
This link and the one referenced in it suggests you use homebrew (brew) to install it
$ ruby <(curl -fsS https://raw.github.com/mxcl/homebrew/go)
$ brew doctor
Remember to add the Homebrew directory to your PATH by adding the directory (found with brew --prefix) to your .bashrc, .zshrc or whatever shell file you’re using (.bashrc is the OS X default). We’ll also add the XQuartz binaries to the PATH in case anything needs them in the future.
export PATH=/usr/local/bin:/opt/X11/bin:$PATH
Start a new Terminal session to pick up the changes.
Now that Homebrew is installed, we can use it to install the required dependencies. Each may take some time as Homebrew generally compiles from source.
$ brew install gfortran # Fortran compiler
$ brew install python # Python interpreter
$ brew install pcre # Regular Expressions library
$ brew install fftw # Fast Fourier Transforms
$ brew install cmake # Cross-platform make
install root
$ brew tap homebrew/science
$ brew install --with-cocoa root
You don't say whether you have installed XCode as well as the commandline tools but I think you will need it
I'm completely new to Homebrew and Anaconda installation and I hope someone could help me out with the warning after executing brew doctor. I'm running on Snow Leopard 10.6.8. The warning is as follows:
Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and what additional flags to use when
compiling and linking.
Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew provided
script of the same name. We found the following "config" scripts:
/Users/user.name/anaconda/bin/freetype-config
/Users/user.name/anaconda/bin/libpng-config
/Users/user.name/anaconda/bin/libpng15-config
/Users/user.name/anaconda/bin/llvm-config
/Users/user.name/anaconda/bin/python-config
/Users/user.name/anaconda/bin/python2-config
/Users/user.name/anaconda/bin/python2.7-config
/Users/user.name/anaconda/bin/xml2-config
/Users/user.name/anaconda/bin/xslt-config
I executed brew --config and the following shows the configuration (hope it helps w/ the issue):
HOMEBREW_VERSION: 0.9.4
ORIGIN: (none)
HEAD: (none)
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CELLAR: /usr/local/Cellar
CPU: dual-core 64-bit penryn
OS X: 10.6.8-i386
Xcode: 3.2.6
GCC-4.0: build 5494
GCC-4.2: build 5666
LLVM-GCC: build 2335
Clang: 1.7 build 77
X11: 2.7.4 => /opt/X11
System Ruby: 1.8.7-358
Perl: /usr/bin/perl
Python: /Users/user.name/anaconda/bin/python
Ruby: /usr/bin/ruby => /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby
Really appreciate your help. Thank you.
tl;dr: you can't really fix this but you can probably ignore it
Those scripts ending in -config provide information for other packages that want to link to them during installation. Let's take libpng for example. If brew (or anything really) compiles a package that depends on libpng it might execute libpng-config to find out some details about the library.
The problem is that brew brings its own version of libpng so if both are installed brew might pick up the wrong libpng-config when installing additional software.
You can do one of four things now:
Ignore the warning if you're confident that the stuff you install with homebrew will not conflict with the stuff that comes with anaconda.
Edit your ~/.bash_profile and remove anaconda from the PATH. If you do this you will have to specify the full path every time you want to run anaconda python.
Move those anaconda config files aside (out of your PATH, e.g. into a config subdirectory). This will likely prevent additional software from linking to anaconda components but should be ok if you intend to keep anaconda isolated.
Remove anaconda completely (just delete the folder) and install brew's version of python. This will also give you pip which should make it easy to reinstall most of the other packages that come with anaconda.
i.e.
brew install python
Then to install e.g. numpy, simply:
pip install numpy
(To make brew's python your default add export PATH="/usr/local/bin:$PATH" to your ~/.bash_profile)
I believe I've found a reasonably safe way to deal with this (if you are using the fish shell).
Install Anaconda but don't add to PATH
Add a symlink to the conda binary somewhere on your PATH (in my case ~/bin)
mkdir ~/bin
cd ~/bin; ln -s ~/anaconda3/bin/conda .
Add section like follows to ~/.config/fish
set -gx PATH $PATH ~/bin
source ~/anaconda3/etc/fish/conf.d/conda.fish
Note that this activate's the conda shell configuration for fish -- use conda activate some-env to switch into an anaconda environment rather than the bash-style source activate some-env. brew doctor should show that this issue no longer exists.
This setup lets you use the conda command to switch into conda-managed environments without worrying about leaky interactions between homebrew-managed python/dependency environments.
The one thing to remember is to avoid running homebrew when a conda environment is active as your path will be set for the conda environmnent and unsafe for homebrew usage.
On arch linux, after install virtualenvwrapper system-wide, via sudo pip2 install virtualenvwrapper and adding this in my user's .bash_profile,
export WORKON_HOME=/home/myuser/.virtualenvs
export PROJECT_HOME=/home/myuser/work
source /usr/bin/virtualenvwrapper.sh
An error shows up whenever I launch a new shell window:-
which: no python in (/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/bin/core_perl)
-bash: : command not found
virtualenvwrapper.sh: There was a problem running the initialization hooks. If Python could not import the module virtualenvwrapper.hook_loader, check that virtualenv has been installed for VIRTUALENVWRAPPER_PYTHON= and that PATH is set properly.
This can be traced to line 50 in the virtualenvwrapper.sh script:-
47 # Locate the global Python where virtualenvwrapper is installed.
48 if [ "$VIRTUALENVWRAPPER_PYTHON" = "" ]
49 then
50 VIRTUALENVWRAPPER_PYTHON="$(\which python)"
51 fi
And is a result of the conflict with arch linux's convention of using python2 for python 2.7 installation.
If I modify line 50 to which python2, everything works perfectly and I will not see the error message whenever I launch a new shell.
What is the appropriate way of resolving this problem? Do I write an explicit check that the current OS is arch linux and introduce an if-else condition to use which python2 in virtualenvwrapper.sh and send the patch to the virtualenvwrapper author? Or is there something I can do with my configuration in my arch linux machine?
Virtualenvwrapper has variables VIRTUALENVWRAPPER_PYTHON and VIRTUALENVWRAPPER_VIRTUALENV which point to your python and virtualenv executables. So, in your .bash_profile you can write something like:
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python2
export VIRTUALENVWRAPPER_VIRTUALENV=/usr/bin/virtualenv2
source /usr/bin/virtualenvwrapper.sh
To answer this arch-specific python quirk (the convention of using python2), I specifically create a soft link like this:-
ln -s /usr/bin/python2 /usr/local/bin/python
Since /usr/local/bin precedes /usr/bin in the bash system PATH environment variable, when I type python, or when the virtualenvwrapper.sh script refers to which python in line 50, we will no longer run into the above mentioned virtualenvwrapper conflict and everything works as expected.
The disadvantage of doing this is that it might mess up the "python 3" (being considered as the default python command) conventions arch linux is designed for. So care has to be taken that I do not use python 3 related libraries or package installations relating to python 3 in my target box.
I found there is no need to edit the /usr/bin/virtualenvwrapper.sh script or create a new symbolic link. I was just missing the virtualenvwrapper module for python2. I installed it like the following:
pip2 install virtualenvwrapper
and keeped the exports so the magic is done.