This question already has answers here:
Using conda activate or specifying python path in bash script?
(2 answers)
Closed 6 months ago.
I am trying to submit a SLURM job on a computing cluster CentOS7. The job is a python file (cifar100-vgg16.py) which needs tensorflow-gpu 2.8.1, which I've installed in a conda environment (tf_gpu). The bash script I'm submitting to SLURM (our job scheduler) is copied below. The SLURM output file shows that the environment being used is the base conda environment Python/3.6.4-foss-2018a (with tensorflow 1.10.1), not tf_gpu. Please advise on how to solve.
Bash script:
#!/bin/bash --login
########## SBATCH Lines for Resource Request ##########
#SBATCH --time=00:10:00 # limit of wall clock time - how long the job will run (same as -t)
#SBATCH --nodes=1 # the number of node requested.
#SBATCH --ntasks=1 # the number of task to run
#SBATCH --cpus-per-task=1 # the number of CPUs (or cores) per task (same as -c)
#SBATCH --mem-per-cpu=2G # memory required per allocated CPU (or core) - amount of memory (in bytes)
#SBATCH --job-name test2 # you can give your job a name for easier identification (same as -J)
########## Command Lines to Run ##########
conda activate tf_gpu
python cifar100-vgg16.py
SLURM output file:
> /opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Tensorflow version 1.10.1
Keras version 2.1.6-tf
Scikit learn version 0.20.0
Traceback (most recent call last):
File "cifar100-vgg16.py", line 39, in <module>
print("Number of GPUs Available:", len(tensorflow.config.experimental.list_physical_devices('GPU')))
AttributeError: module 'tensorflow' has no attribute 'config'
There is a mistake in your job script. Replace conda activate tf_gpu with source activate tf_gpu.
Also, I guess you need to load the module so that you can use it. It would be something like module load anaconda check module avail for the list of available modules.
But looks like you HPC doesn't need a module load as it identifies conda without module load.
EDIT: FlytingTeller said that source activate is replaced with conda activate in 2017. I know this.
I don't know if this works on HPCs. To prove my point here is the output of Swansea SUNBIRD, when I use conda activate.
(base) hell#Dell-Precision-T7910:~$ ssh sunbird
Last login: Wed Aug 10 15:30:29 2022 from en003013.swan.ac.uk
====================== Supercomputing Wales - Sunbird ========================
This system is for authorised users, if you do not have authorised access
please disconnect immediately, and contact Technical Support.
-----------------------------------------------------------------------------
For user guides, documentation and technical support:
Web: http://portal.supercomputing.wales
-------------------------- Message of the Day -------------------------------
SUNBIRD has returned to service unchanged. Further information on
the maintenance outage and future work will be distributed soon.
===============================================================================
[s.1915438#sl2 ~]$ module load anaconda/3
[s.1915438#sl2 ~]$ conda activate base
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
If your shell is Bash or a Bourne variant, enable conda for the current user with
$ echo ". /apps/languages/anaconda3/etc/profile.d/conda.sh" >> ~/.bashrc
or, for all users, enable conda with
$ sudo ln -s /apps/languages/anaconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh
The options above will permanently enable the 'conda' command, but they do NOT
put conda's base (root) environment on PATH. To do so, run
$ conda activate
in your terminal, or to put the base environment on PATH permanently, run
$ echo "conda activate" >> ~/.bashrc
Previous to conda 4.4, the recommended way to activate conda was to modify PATH in
your ~/.bashrc file. You should manually remove the line that looks like
export PATH="/apps/languages/anaconda3/bin:$PATH"
^^^ The above line should NO LONGER be in your ~/.bashrc file! ^^^
[s.1915438#sl2 ~]$ source activate base
(base) [s.1915438#sl2 ~]$
Here is the output of Cardiff HAWK when I use conda activate.
(base) hell#Dell-Precision-T7910:~$ ssh cardiff
Last login: Tue Aug 2 09:32:44 2022 from en003013.swan.ac.uk
======================== Supercomputing Wales - Hawk ==========================
This system is for authorised users, if you do not have authorised access
please disconnect immediately, and contact Technical Support.
-----------------------------------------------------------------------------
For user guides, documentation and technical support:
Web: http://portal.supercomputing.wales
-------------------------- Message of the Day -------------------------------
- WGP Gluster mounts are now RO on main login nodes.
- WGP RW access is via Ser Cymru system or dedicated access VM.
===============================================================================
[s.1915438#cl1 ~]$ module load anaconda/
anaconda/2019.03 anaconda/2020.02 anaconda/3
anaconda/2019.07 anaconda/2021.11
[s.1915438#cl1 ~]$ module load anaconda/2021.11
INFO: To setup environment run:
eval "$(/apps/languages/anaconda/2021.11/bin/conda shell.bash hook)"
or just:
source activate
[s.1915438#cl1 ~]$ conda activate
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
[s.1915438#cl1 ~]$ conda activate base
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
[s.1915438#cl1 ~]$ source activate base
(2021.11)[s.1915438#cl1 ~]$
The conda versions are certainly after 2020 not 2017. SInce, the question was about a HPC cluster, that I why said replace conda activate with source activate, to activate the first conda environment.
Anyone with a possible explanation?
EDIT 2: I think I have an explanation.
[s.1915438#sl2 ~]$ cat ~/.bashrc
# .bashrc
# Dynamically generated by: genconfig (Do not edit!)
# Source global definitions
if [ -f /etc/bashrc ]; then
. /etc/bashrc
fi
# Load saved modules
module load null
# Personal settings file
if [ -f $HOME/.myenv ]
then
source $HOME/.myenv
fi
The ~/.bashrc does not contain the path to conda.sh. I think this is true for many HPCs.
I'm making an installer using the below bash script. After activating the env and later on checking the active env using the conda info, it shows that No env is active. Please refer to the image
Even after installing dependencies, the packages are not installed in the env, instead it's getting installed in the base env.
The script-
#!/usr/bin/bash
set -euo pipefail
conda create --name count python=3.7 <<< 'y'
. $(conda info --base)/etc/profile.d/conda.sh
conda activate count
conda info
#install dependencies
pip install -r requirements_count.txt
Thanks in advance for going through the query!
First, iteratively using ad hoc installs is a bad pattern. Better practice is to predefine everything in the environment with a YAML. That can even include PyPI packages and requirements.txt files.
count.yaml
name: count
channels:
- conda-forge
dependencies:
## Conda deps
- python=3.9 ## change to what you want, but preferable to specify
## PyPI deps
- pip
- pip:
- -r requirements_count.txt
Second, conda activate is intended for interactive shells, not scripts. One can run a bash script interactively by including an -l flag in the shebang. But better practice is to use the conda run method, which is designed for programmatic execution of commands in a specified context.
Pulling this together, the script might be:
script.sh
#!/usr/bin/bash
set -euo pipefail
## create the environment
conda create -n count -f count.yaml
## execute a Python script in the environment
conda run -n count python my_code.py
If the code run in the environment needs to interact with a user or stream output, you may need additional flags (--no-capture-output, --live-stream). See conda run --help.
I would like to change my conda environment from a bash script.
I want to run bash script_yxz, where 'script_xyz' is like:
#!/bin/bash
conda activate my_env
and switch to my_env.
This already works if I run source script_yxz.
But I have the problem that I am not able to 'source' on remote machines with 'sshpass'.
To better understand my purposes, my goal is to run on my terminal
sshpass -p "password" ssh -o user#server "bash script_xyz"
and changing the environment on the server.
This is why I need to use bash instead of source.
I have read a lot of solutions on various forums but none of them works.
seems like the conda script is not imported by default so this should fix it
source ~/anaconda3/etc/profile.d/conda.sh
conda activate <env>
#!/bin/bash
eval "$(conda shell.bash hook)"
source ~/anaconda3/etc/profile.d/conda.sh
conda create -n testing python=3.10 -y
conda activate testing
python --version
#output
You can use bash,zsh or any shell aliases for this purposes. You just add
alias my_conda='source /home/$USER/anaconda3/bin/activate && conda activate MyPy38'
line into the .bashrc,.zshrc or .any_other_shell_rc.
"N.B. My environment name is MyPy38". So,replace it according name as well as the path /home/$USER/anaconda3.
Also you can create separate file for aliases. Just create a file called .bash_aliases and add
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
lines to .bashrc,.zshrc or .any_other_shell_rc and keep the command
alias my_conda='source /home/$USER/anaconda3/bin/activate && conda activate MyPy38'
into the .bash_aliases. Now, source ~/.zshrc ~/.bashrc or just close and open a new terminal. Run the command my_conda and BOOM!
Also, you can add some other aliases for jupyter-notebook jupyter-lab spyder etc. like
# Just activate my conda
alias my_conda='source /home/$USER/anaconda3/bin/activate && conda activate MyPy38'
# Open Jupyter Notebook in my Env
alias my_jupn='source /home/$USER/anaconda3/bin/activate && conda activate MyPy38 && jupyter-notebook'
# Open Jupyter Lab in my Env
alias my_jupl='source /home/$USER/anaconda3/bin/activate && conda activate MyPy38 && jupyter-lab'
# Open Spyder in my Env
alias my_spyder='source /home/$USER/anaconda3/bin/activate && conda activate MyPy38 && spyder'
To confirm active environment name python code
import sys
print(sys.executable)
I am using conda 4.6.8 to test a python package in a conda env on Travis CI. I want to replace my old source activate ENVNAME line with the new conda activate ENVNAME command in my Travis CI configuration. If I run this on Travis:
>>> conda update -n base conda
>>> conda init
no change /home/travis/miniconda/condabin/conda
no change /home/travis/miniconda/bin/conda
no change /home/travis/miniconda/bin/conda-env
no change /home/travis/miniconda/bin/activate
no change /home/travis/miniconda/bin/deactivate
no change /home/travis/miniconda/etc/profile.d/conda.sh
no change /home/travis/miniconda/etc/fish/conf.d/conda.fish
no change /home/travis/miniconda/shell/condabin/Conda.psm1
no change /home/travis/miniconda/shell/condabin/conda-hook.ps1
no change /home/travis/miniconda/lib/python3.7/site-packages/xonsh/conda.xsh
no change /home/travis/miniconda/etc/profile.d/conda.csh
modified /home/travis/.bashrc
==> For changes to take effect, close and re-open your current shell. <==
How can I "close and re-open" my shell on Travis? Because otherwise I cannot activate my conda environment:
>>> conda create -n TEST package_names
>>> conda activate TEST
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
The command "conda activate TEST" failed and exited with 1 during .
Your build has been stopped.
Not sure it is currently supported as the official doc still uses source in travis.yml.
What does conda init do?
This new command should harmonize the way users setup their shells to be able to call conda activate.
Actually, if you run conda init --dry-run --verbose you will see that it tries to source conda.sh from your ~/.bashrc (assuming you're running Bash, from info mentioned in your question).
And conda.sh will define a conda() function that will catch a few commands among which activate and deactivate and dispatch to $CONDA_EXE:
conda() {
if [ "$#" -lt 1 ]; then
"$CONDA_EXE"
else
\local cmd="$1"
shift
case "$cmd" in
activate|deactivate)
__conda_activate "$cmd" "$#"
;;
install|update|upgrade|remove|uninstall)
"$CONDA_EXE" "$cmd" "$#" && __conda_reactivate
;;
*) "$CONDA_EXE" "$cmd" "$#" ;;
esac
fi
}
So unless this function is defined in your local shell, you won't be able to call conda activate.
Hint on a solution? (not tested for Travis CI)
The only hint I can suggest is to try source $(conda info --root)/etc/profile.d/conda.sh and then conda activate. This should do roughly the same as conda init assuming you are using Bourne shell derivatives.
For csh there is $(conda info --root)/etc/profile.d/conda.csh, and for fish there is $(conda info --root)/etc/fish/conf.d/conda.fish
Note: although not tested for Travis CI, this solution works for me from bash. Of course, the conda executable should be found in PATH for conda info --root to work properly.
I am using vagrant (1.9.1 on MacOSX Sierra) to provision ubuntu/xenial64 box on VirtualBox to run a python app. I am unable to activate a conda environment using the normal shell command source while provisioning. In my bootstrap.sh, I have the following lines for creation of a new environment and then switching to it.
#!/usr/bin/env bash
set -e # Exit script immediately on first error.
set -x # Print commands and their arguments as they are executed.
/home/ubuntu/miniconda3/bin/conda create --name envmycondaenvironment python=3.5 # environment with python3.5
source activate envgatherurls
I receive the following error from vagrant.
==> default: + source activate envmycondaenvironment
==> default: /tmp/vagrant-shell: line 21: activate: No such file or directory
Why is activate not found by the shell script? I verified that /home/ubuntu/miniconda3/bin/ where activate can be found has been added to PATH in the .bashrc file.
The command activate is provided by conda and is not automatically added to the PATH environment variable. Please note that the bootstrap.sh script runs as root and not the vagrant user. So you need to make sure that the .bashrc for the root user has /home/ubuntu/miniconda3/bin in it's path. If I were you, I had rather do this:
#!/usr/bin/env bash
set -e # Exit script immediately on first error.
set -x # Print commands and their arguments as they are executed.
export PATH=/home/ubuntu/miniconda3/bin:$PATH
conda create --name envmycondaenvironment python=3.5 # environment with python3.5
source activate envgatherurls