How to pip install into a virtualenv via a bash install script? - bash

I recently tried to automate the setup of an Ubuntu VM with a bash script (I am new to bash scripting).
The issue is that the way I set it up, it doesn't work. Particularly the mkvirtualenv and workon commands don't work in the bash script. How do I create a virtualenv in a bash script with passing it a variable and then install into the virtualenv via pip?
#!/bin/bash
VENV_NAME='name_of_virtualenv'
#Setting up virtualenv
mkdir --mode=770 /var/virtualenvs
chown -R www-data:www-edit /var/virtualenvs
chmod 771 /var/virtualenvs
echo '# virtualenv and virtualwrapper' >> ~/.bashrc
echo ' export VIRTUALENV_USE_DISTRIBUTE=1' >> ~/.bashrc # <-- Always use pip/distribute
echo ' export WORKON_HOME=/var/virtualenvs' >> ~/.bashrc
echo ' source /usr/local/bin/virtualenvwrapper.sh' >> ~/.bashrc
echo ' export PIP_VIRTUALENV_BASE=$WORKON_HOME' >> ~/.bashrc
echo ' export PIP_RESPECT_VIRTUALENV=true' >> ~/.bashrc
source ~/.bashrc
mkvirtualenv --distribute '{VENV_NAME}'
workon {VENV_NAME}
pip install psycopg2
pip install --upgrade PIL

You missing dollar before the call, also did not see an export for VENV_NAME
export VIRTUALENV_USE_DISTRIBUTE=1
echo {VIRTUALENV_USE_DISTRIBUTE}
{VIRTUALENV_USE_DISTRIBUTE}
echo ${VIRTUALENV_USE_DISTRIBUTE}
1
Unsure why you need to export out to bashrc and from the looks of it each time you run it, it would add the same exports to bashrc which will end up with a larger and larger bashrc file each time
Why not just make them local variables like
VIRTUALENV_USE_DISTRIBUTE=1
workon $VIRTUALENV_USE_DISTRIBUTE

Related

Makefile not exporting env var to submake unless defined in the same recipe

Please see the below Makefile:
SHELL=/bin/bash -euo pipefail
REPO_ROOT = $(shell pwd)
export VIRTUAL_ENV := ${REPO_ROOT}/venv
# bin = POSIX, Scripts = Windows
export PATH := ${VIRTUAL_ENV}/bin:${VIRTUAL_ENV}/Scripts:${PATH}
show-python: ## Show path to python and version
#echo -n "python location: "
#python -c "import sys; print(sys.executable, end='')"
#echo -n ", version: "
#python -c "import platform; print(platform.python_version())"
install: show-python
install: ## Install all dev dependencies into a local virtual environment.
export VIRTUAL_ENV="${VIRTUAL_ENV}"; \
python -m pip install -r requirements-dev.txt --progress-bar off
The install recipe only works if I manually export the environment variable VIRTUAL_ENV inside the recipe with ; chaining it into the next command. In other words, the first line of the install recipe is currently "doing something special" that I can't figure out.
What is going on here with this export being required twice?
The export of bash and the export of Make are not quite the same thing.
The former marks a bash variable to be passed to any child processes started later with fork. The latter marks a Make variable to be passed to any sub-Make launched later.
Your makefile does not use recursive make; that is, it does not launch any sub-Make. So your first export does nothing and can be removed.
(I am the OP).
MINGW64: Didn't Work
Per the comments, I updated the Makefile recipe:
install: show-python
install: ## Install all dev dependencies into a local virtual environment.
echo $$VIRTUAL_ENV; echo $$PATH; echo $$SHELL; echo $$BASH_VERSION
python -m pip install -r requirements-dev.txt --progress-bar off \
--extra-index-url $(PIP_EXTRA_INDEX_URL)
It prints the below, where everything looks correct:
echo $VIRTUAL_ENV; echo $PATH; echo $SHELL; echo $BASH_VERSION
/c/path/to/repo/venv
/c/path/to/repo/venv/bin:/c/path/to/repo/venv/Scripts:/c/Users/user/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/usr/bin:/mingw64/bin:/usr/bin:/c/Users/user/bin:/c/Users/user/.pyenv/pyenv-win/bin:/c/Users/user/.pyenv/pyenv-win/shims:/c/Python38/Scripts:/c/Python38:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/WINDOWS/System32/OpenSSH:/c/ProgramData/chocolatey/bin:/c/Program Files/dotnet:/cmd:/c/Users/user/.pyenv/pyenv-win/bin:/c/Users/user/.pyenv/pyenv-win/shims:/c/Users/user/AppData/Local/Microsoft/WindowsApps:/usr/bin/vendor_perl:/usr/bin/core_perl
/usr/bin/bash.exe
5.1.16(1)-release
Note that I am using Git Bash from Git 2.38.1.windows.1 (Git for Windows) with MINGW64 (installed via msys2-runtime 3.3.6-1).
Cygwin: Works
Next, I was like maybe it's a MINGW64 issue, so let me try Cygwin (version 3.3.6-341.x86_64 which uses bash version 4.4.12(3)-release).
Sure enough, it works out the box, no need for the pass-through export VIRTUAL_ENV="${VIRTUAL_ENV}"; \.
Conclusions
Use Cygwin instead of Git Bash
I opened an issue with MINGW64: https://github.com/mingw-w64/mingw-w64/issues/16

pyenv - environment "activated", but python and pip not found

I suppose there is something wrong with my bash init scripts (like .bashrc or .bash_profile). But let's start from beginning.
I can create and activate pyenv environment, but when I try to use python, I get error: -bash: python: command not found.
It looks like pyenv understands creation and swapping envorinments. I mean, it's probably not malformed. There is preview of my tries:
$ mkdir test-python-project
$ cd test-python-project/
$ pyenv versions
* system (set by /home/vagrant/.pyenv/version)
3.7.10
3.7.10/envs/k-pkb-env
$ pyenv virtualenv 3.7.10 test-env
Looking in links: /tmp/tmpkwojcc1e
Requirement already satisfied: setuptools in /home/vagrant/.pyenv/versions/3.7.10/envs/test-env/lib/python3.7/site-packages (47.1.0)
Requirement already satisfied: pip in /home/vagrant/.pyenv/versions/3.7.10/envs/test-env/lib/python3.7/site-packages (20.1.1)
$ pyenv activate test-env
pyenv-virtualenv: prompt changing will be removed from future release. configure export PYENV_VIRTUALENV_DISABLE_PROMPT=1 to simulate the behavior.
(test-env) $ python
-bash: python: command not found
(test-env) $ pyenv local test-env
(test-env) $ cd ..
(test-env) $ pyenv deactivate
$ cd test-python-project/
(test-env) $ python
-bash: python: command not found
(test-env) $ pip
-bash: pip: command not found
(test-env) $ pyenv version
test-env (set by /home/vagrant/Work/test-python-project/.python-version)
I'm not sure how to configure bash init scripts, because in pyenv readme they suggest using .profile, which I don't have.
So, there are my bash inits:
.bashrc
$ cat .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
.bash_profile
$ cat .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
# PyEnv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
Some additional information:
$PATH variable
$ echo $PATH
/home/vagrant/.pyenv/plugins/pyenv-virtualenv/shims:/home/vagrant/.pyenv/bin:/home/vagrant/.local/bin:/home/vagrant/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
It's a bit strange for me, because this additional paths added by pyenv doesn't seem to contain path to desired virtual environment:
$ ls /home/vagrant/.pyenv/plugins/pyenv-virtualenv/shims
activate deactivate
$ ls /home/vagrant/.pyenv/bin
pyenv
type python
$ type python
-bash: type: python: not found
which python
$ which python
/usr/bin/which: no python in (/home/vagrant/.pyenv/plugins/pyenv-virtualenv/shims:/home/vagrant/.pyenv/bin:/home/vagrant/.local/bin:/home/vagrant/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
I tried also pyenv rehash, but also still no effect:
(test-env) [vagrant#centos test-python-project]$ pyenv rehash
(test-env) [vagrant#centos test-python-project]$ python
-bash: python: command not found
With some help from #Simba, I managed to have my configuration correct:
.bash_profile
# .bash_profile
# !!! ITS IMPORTANT THESE LINES MUST BE BEFORE . ~/.bashrc
# PyEnv - only path-related
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
# !!! ITS IMPORTANT THESE LINES ABOVE MUST BE BEFORE . ~/.bashrc
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
.bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# User specific environment
if ! [[ "$PATH" =~ "$HOME/.local/bin:$HOME/bin:" ]]
then
PATH="$HOME/.local/bin:$HOME/bin:$PATH"
fi
export PATH
# Uncomment the following line if you don't like systemctl's auto-paging feature:
# export SYSTEMD_PAGER=
# User specific aliases and functions
# PyEnv - commands
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
Solution
READ THE PYENV GUIDE CAREFULLY.
You didn't follow pyenv's README guide correctly. The guide tells you put PATH related operation in .bash_profile or .profile. But eval "$(pyenv init -)" in .bashrc.
Move pyenv init script from .bash_profile to .bashrc.
# Put it in .bashrc
# PyEnv
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
Extended Reading
You bash shell is not a login shell. .bash_profile is not sourced at all, which skip pyenv init -.
Bash initialization
login mode:
/etc/profile
~/.bash_profile, ~/.bash_login, ~/.profile (only the first one that exists)
interactive non-login:
/etc/bash.bashrc (some Linux; not on Mac OS X)
~/.bashrc
non-interactive:
source file in $BASH_ENV
The default shell on Linux is a non-login, interactive shell. The default shell on macOS is a login, interactive shell.
There's also a detailed explanation about the shell startup by flowblok
Ref
Unix shell initialization from pyenv wiki
Shell startup scripts

export RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl#1.1)"eval "$(rbenv init -)" not valid in this context: export path

I was trying to install ruby on my machine. I had to add this -
export RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl#1.1)"eval "$(rbenv init -)"
The error I get every time I open my terminal
/Users/akash/.zshrc:export:23: not valid in this context: export PATH
Rest of my .zshrc -
# The next line updates PATH for the Google Cloud SDK.
if [ -f '/Users/akash/Desktop/Projects/cheaplops/clear/google-cloud-sdk/path.zsh.inc' ]; then . '/Users/akash/Desktop/Projects/cheaplops/clear/google-cloud-sdk/path.zsh.inc'; fi
# The next line enables shell command completion for gcloud.
if [ -f '/Users/akash/Desktop/Projects/cheaplops/clear/google-cloud-sdk/completion.zsh.inc' ]; then . '/Users/akash/Desktop/Projects/cheaplops/clear/google-cloud-sdk/completion.zsh.inc'; fi
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/Users/akash/anaconda3/bin/conda' 'shell.zsh' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/Users/akash/anaconda3/etc/profile.d/conda.sh" ]; then
. "/Users/akash/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/Users/akash/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<
export RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl#1.1)"eval "$(rbenv init -)"
export PATH=${PATH}:/usr/local/mysql/bin
First check do you have openssl installed, the second part is for rbenv but first one is setting openssl so your software can use that specific version.
brew info openssl#1.1
See what it show will give you details or not installed. If not installed you have to install it. Then run following command to see you can see directory from brew for openssl.
brew --prefix openssl#1.1
If path is visible that mean all good. Now check do you have rbenv install? that is software to be installed before doing this.
brew install rbenv
Now I found one error in your whole line, as that line is actually creating export entry, but there is nothing to separate command eval from export, try following command.
export RUBY_CONFIGURE_OPTS="--with-openssl-dir=$(brew --prefix openssl#1.1)"; eval "$(rbenv init -)"
If you notice before eval I added ; so one process complete and next can run. You are basically setting options --with-open... in a environment variable, and then you are running command of rbenv. You can use following lines to install ruby I took reference from rbenv discussion
RUBY_CONFIGURE_OPTS="--with-openssl-dir=/usr/local/opt/openssl"; rbenv install 2.3.8

How can I avoid typing the "source ~/.bash_profile" command in the terminal every time?

I just installed Anaconda. When I try to run a command, lets say conda --version, I get the following error message:
zsh: command not found: conda
When I type in:
source ~/.bash_profile
Everything works:
conda --version
conda 4.8.0
But I have to type the command every time I start the terminal.
My .bash_profile looks like this:
# added by Anaconda3 2019.10 installer
# >>> conda init >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$(CONDA_REPORT_ERRORS=false '/Users/myusername/opt/anaconda3/bin/conda' shell.bash hook 2> /dev/null)"
if [ $? -eq 0 ]; then
\eval "$__conda_setup"
else
if [ -f "/Users/myusername/opt/anaconda3/etc/profile.d/conda.sh" ]; then
. "/Users/myusername/opt/anaconda3/etc/profile.d/conda.sh"
CONDA_CHANGEPS1=false conda activate base
else
\export PATH="/Users/myusername/opt/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda init <<<
What can I do to fix this?
System: macOS
place source ~/.bash_profile into ~/.zsh
Bash loads .bash_profile during init, for zsh it’s .zsh
Default shell was changed from bash to zsh with macOS Catalina, so that’s why there is the problem.

Why my file does not get sourced from bash script?

I have a bash script where at some point, I want to source the ${HOME}/.profile file which should add ${HOME}/.local/bin to the $PATH. But when I check the path with echo $PATH, ${HOME}/.local/bin is absent as if the source did not happen. What am I doing wrong?
if command -v pip3 &>/dev/null; then
echo "Pip is already installed."
else
echo "Pip is not installed. Installing Pip..."
cd ${HOME}/Downloads
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
su -c "python3 get-pip.py --user" "$SUDO_USER"
cat <<-'EOT' >> "${HOME}/.profile"
# set PATH so it includes user's private .local/bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
PATH="$HOME/.local/bin:$PATH"
fi
EOT
source "${HOME}/.profile" #this is not happening!!!
rm ${HOME}/Downloads/get-pip.py
echo "Pip has been installed."
fi
Thanks in advance.
EDIT: Fixed the script syntax as suggest by Kusalananda.
A script can't modify the environment of the shell from whence it was executed.
Sourcing ~/.profile in the script will not set the path in the interactive shell that originally started the script. To do that, you would have to source your script.
Also, your here-document would need to be quoted, or the current values of HOME and PATH would be inserted into the .profile file:
cat <<'PROFILE_END' >> "$HOME/.profile"
# set PATH so it includes user's private .local/bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
PATH="$HOME/.local/bin:$PATH"
fi
PROFILE_END
Also note that if the user is a bash user with an existing ~/.bash_profile file, then the ~/.profile file will be ignored for that user when a new login shell is started.
I'm further unsure why you su to $USER. It seems like an unnecessary step.

Resources