I am hoping to run a simple shell script to ease the management around some conda environments. Activating conda environments via conda activate in a linux os works fine in the shell but is problematic within a shell script. Could someone point me into the right direction as to why this is happening?
Example to repeat the issue:
# default conda env
$ conda info | egrep "conda version|active environment"
active environment : base
conda version : 4.6.9
# activate new env to prove that it works
$ conda activate scratch
$ conda info | egrep "conda version|active environment"
active environment : scratch
conda version : 4.6.9
# revert back to my original conda env
$ conda activate base
$ cat shell_script.sh
#!/bin/bash
conda activate scratch
# run shell script - this will produce an error even though it succeeded above
$ ./shell_script.sh
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
I use 'source command' to run the shell script, it works:
source shell_script.sh
The error message is rather helpful - it's telling you that conda is not properly set up from within the subshell that your script is running in. To be able to use conda within a script, you will need to (as the error message says) run conda init bash (or whatever your shell is) first. The behaviour of conda and how it's set up depends on your conda version, but the reason for the version 4.4+ behaviour is that conda is dependent on certain environment variables that are normally set up by the conda shell itself. Most importantly, this changelog entry explains why your conda activate and deactivate commands no longer behave as you expect in versions 4.4 and above.
For more discussion of this, see the official conda issue on GitHub.
Edit: Some more research tells me that the conda init function mentioned in the error message is actually a new v4.6.0 feature that allows a quick environment setup so that you can use conda activate instead of the old source activate. However, the reason why this works is that it adds/changes several environment variables of your current shell and also makes changes to your RC file (e.g.: .bashrc), and RC file changes are never picked up in the current shell - only in newly created shells. (Unless of course you source .bashrc again). In fact, conda init --help says as much:
IMPORTANT: After running conda init, most shells will need to be closed and restarted for changes to take effect
However, you've clearly already run conda init, because you are able to use conda activate interactively. In fact, if you open up your .bashrc, you should be able to see a few lines added by conda teaching your shell where to look for conda commands. The problem with your script, though, lies in the fact that the .bashrc is not sourced by the subshell that runs shell scripts (see this answer for more info). This means that even though your non-login interactive shell sees the conda commands, your non-interactive script subshells won't - no matter how many times you call conda init.
This leads to a conjecture (I don't have conda on Linux myself, so I can't test it) that by running your script like so:
bash -i shell_script.sh
you should see conda activate work correctly. Why? -i is a bash flag that tells the shell you're starting to run in interactive mode, which means it will automatically source your .bashrc. This should be enough to enable you to use conda within your script as if you were using it normally.
Quick solution for bash: prepend the following init script into your Bash scripts.
eval "$(command conda 'shell.bash' 'hook' 2> /dev/null)"
Done.
For other shells, check the init conf of your shell, copy the following content within the shell conf and prepend it into your scripts.
# >>> conda initialize >>>
...
# <<< conda initialize <<<
You can also use
conda init --all --dry-run --verbose
to get the init script you need in your scripts.
Explanation
This is related with the introduction of conda init in conda 4.6.
Quote from conda 4.6 release log
Conda 4.4 allowed “conda activate envname”. The problem was that setting up your shell to use this new feature was not always straightforward. Conda 4.6 adds extensive initialization support so that more shells than ever before can use the new “conda activate” command. For more information, read the output from “conda init –help”
After conda init is introduced in conda 4.6, conda only expose command
conda into the PATH but not all the binaries from "base". And environment switch is unified by conda activate env-name and conda deactivate on all platforms.
But to make these new commands work, you have to do an additional initialization with conda init.
The problem is that your script file is run in a sub-shell, and conda is not initialized in this sub-shell.
References
Conda 4.6 Release
Unix shell initialization
Shell startup scripts
Using conda activate or source activate in shell scripts does not always work and can throw errors like this. An easy work around it to place source ~/miniconda3/etc/profile.d/conda.sh above any conda activate command in the script:
source ~/miniconda3/etc/profile.d/conda.sh # Or path to where your conda is
conda activate some-conda-environment
This is the solution that has worked for me and will also work if sharing scripts. This also gets around having to use conda init as on some clusters I have worked with the system is initialised but conda activate still won't work in a shell script.
if you want to use the shell script to run the other python file in the other conda env, just run the other file via the following command.
os.system('conda run -n <env_name> python <path_to_other_script>')
What is the problem with simply doing something like this in your shell:
source /opt/conda/etc/profile.d/conda.sh
(The conda init is still marked as Experimental, and thus not sure if it is a good idea to use it yet).
I also had the exact same error when trying to activate conda env from C++ or Python file. I solved it by bypassing the conda activate statement and using the absolute path of the specific conda env.
For me, I set up an environment called "testenv" using conda.
I searched all python environments using
whereis python | grep 'miniconda'
It returned a list of python environments. Then I ran my_python_file.py using the following command.
~/miniconda3/envs/testenv/bin/python3.8 my_python_file.py
You can do the same thing on windows too but looking up for python and conda python environments is a bit different.
This answer from Github worked for me (I'm using Ubuntu so it's not for Windows only):
eval "$(conda shell.bash hook)"
conda activate my_env
I just followed a similar solution like the one from hong-xu
So to run a shell command that calls the script with arguments and using a specific conda environment:
from a jupyter cell, goes like this :
p1 = <some-value>
run = f"conda run -n {<env-name>} python {<script-name>.py} \
--parameter_1={p1}"
!{run}
I didn't find any of the above scripts useful. These are fine if you want to run conda in non-interactive mode, but I'd like to run it in interactive mode. If I run:
conda activate my_environment
in a bash script it just runs in the script.
I found that creating an alias in .bashrc is all that is required to change directory to a particular project I'm working on, and set up the correct conda environment for me. So I included in .bashrc:
alias my_environment="cd ~/subdirectory/my_project && conda activate my_environment"
and then:
source ~/.bashrc
Then I can just type at the command line:
my_environment
to change to the correct project and correct environment everytime I want to work on a different project.
This answer is similar to #Lamma answer. This worked for me ->
(1) I defined several variables; the conda activate function, environments directory and environment
conda_activate=~/anaconda3/bin/activate
conda_envs_dir=~/anaconda3/envs
conda_env=<env name>
(2) I source conda activate with the environment
source ${conda_activate} ${conda_envs_dir}/${conda_env}
(3) then you can run your python script
python <path to script.py>
This bypasses the conda init requirement. my .bashrc already was initialized and sourcing the .bashrc file didn't work for me. #Lamma's answer worked for me as well as the above code.
The problem is that when you run the bash script, a new (linux) shell environment is created that was not initialized properly. If your intention is to activate a conda environment, and then run python through the script, you can properly initialize the created shell
environment as discussed in the accepted solution.
If however you want to have the conda environment active after you finish this script, then this will not work because the conda environment has changed on the new shell and you exit that shell when you finish the script. Think of this as running bash, then conda activate... then running exit to exit that bash... More details in How to execute script in the current shell on Linux?:
TL;DR:
Add the line #!/bin/bash as the first line of the script
Type the command source shell_script.sh or . shell_script.sh
Note: . in bash is equivalent to source in bash.
$ conda activate scratch
or
$ source activate scratch
#open terminal or CMD as administrator
$ cd <path Anaconda3 install>\Scripts
$ activate
$ cd ..
$ conda activate scratch
I'm trying to activate a conda environment on a cluster. However, I keep being returned with the shell needing to be initialized. Ruuning conda init bash does nothing and even when closing the shell and trying again I get the same 'conda is not initialized'?
-bash-4.2$ conda activate libraries/
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
-bash-4.2$ conda init bash
no change /usr/local/miniconda3/condabin/conda
no change /usr/local/miniconda3/bin/conda
no change /usr/local/miniconda3/bin/conda-env
no change /usr/local/miniconda3/bin/activate
no change /usr/local/miniconda3/bin/deactivate
no change /usr/local/miniconda3/etc/profile.d/conda.sh
no change /usr/local/miniconda3/etc/fish/conf.d/conda.fish
no change /usr/local/miniconda3/shell/condabin/Conda.psm1
no change /usr/local/miniconda3/shell/condabin/conda-hook.ps1
no change /usr/local/miniconda3/lib/python3.8/site-packages/xontrib/conda.xsh
no change /usr/local/miniconda3/etc/profile.d/conda.csh
no change /nethome/6966225/.bashrc
No action taken.
-bash-4.2$ conda activate libraries/
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
The conda init command only updates the shell initialization files (e.g., .bashrc) to include shell function setup. One still needs to source the relevant initialization file. Typically, this is done by restarting the shell session or running . ~/.bashrc.
Be aware that the shell also needs to be launched in a way that it loads the relevant initialization file. Most user-initiated shell sessions do this automatically, but in some situations one may need to explicitly include a flag for interactive (-i in bash) or login (-l in bash) sessions to ensure loading of .bashrc or .bash_profile files.
I have problem with refactoring bash command into tcsh-friendly command. Don't know the tcsh syntax very well so the error im receiving doens't give me any clue.
The whole bash script was about adding modules on server
MODULEPATH=/app/modules/0/modulefiles:/env/common/modules
export MODULEPATH
module() { eval `/app/modules/0/bin/modulecmd sh "$#"` ;}
I changed first two commands to tcsh already
setenv MODULEPATH /app/modules/0/modulefiles:/env/common/modules
set MODULEPATH
But i dont know how to change the syntax of last command. Console is returning me error "Badly placed ()'s.".
Can I ask for little lesson what to change in this command to be tcsh-friendly?
chepner is right saying tcsh doesn't have functions at all and you'd have to write an alias instead. That's not much of an effort for your one-line function:
alias module 'eval `/app/modules/0/bin/modulecmd sh \!*`'
Basically, we prepend alias, remove () and {}, quote and replace "$#" with \!*.
The module command you want to port from bash to tcsh already comes with an initialization file to the tcsh shell. For all shells or scripting languages module is compatible with, there is an initialization file provided in the init directory of the software.
So from your example, Modules is installed in /app/modules/0, so you should have a /app/modules/0/init directory and a /app/modules/0/init/tcsh script to initialize the module command for tcsh. You just have to source it to initialize the module command:
source /app/modules/0/init/tcsh
As Armali says, the module command on tcsh is defined with the alias shell command.
With recent version of Modules (version 4+), you also have the possibility to define the module command in your current shell session with the autoinit subcommand of the modulecmd script:
eval `/app/modules/0/bin/modulecmd tcsh autoinit`
I own the licensed version of Pycharm. I have installed BashSupport plugin as well.
However, I am not sure what steps are required to remotely test and execute a bash script on RHEL server.
I am able to configure a remote interpreter but the interpreter path is /usr/bin/python.
In my logic you need to change interpreter path to /bin/bash instead of /usr/bin/python or i don't understand what you are trying to do..
I've met the save problem today.
If I use the default pycharm bash configuration, I'll get the following error:
C:\windows\system32\bash.exe C:/git/NewProject/run.sh
/bin/bash: C:/git/NewProject/run.sh: No such file or directory
Process finished with exit code 127
So, I change the default Windows Script path to Linux absolute Script path as following:
Run successfully.
You gotta check the path on server, If there is alias set, You can check alias
$ which python
$ whence python (Some version)
$ ls -l `which python`
I tried activating a VirtualEnv through a shell script like the one below but it doesn't seem to work,
#!/bin/sh
source ~/.virtualenvs/pinax-env/bin/activate
I get the following error
$ sh virtualenv_activate.sh
virtualenv_activate.sh: 2: source: not found
but if I enter the same command on terminal it seems to work
$ source ~/.virtualenvs/pinax-env/bin/activate
(pinax-env)gautam#Aspirebuntu:$
So I changed the shell script to
#!/bin/bash
source ~/.virtualenvs/pinax-env/bin/activate
as suggested and used
$ bash virtualenv_activate.sh
gautam#Aspirebuntu:$
to run the script .
That doesn't throw an error but neither does that activate the virtual env
So any suggestion on how to solve this problem ?
PS : I am using Ubuntu 11.04
TLDR
Must run the .sh script with source instead of the script solely
source your-script.sh
and not
your-script.sh
Details
sh is not the same as bash (although some systems simply link sh to bash, so running sh actually runs bash). You can think of sh as a watered down version of bash. One thing that bash has that sh does not is the "source" command. This is why you're getting that error... source runs fine in your bash shell. But when you start your script using sh, you run the script in an shell in a subprocess. Since that script is running in sh, "source" is not found.
The solution is to run the script in bash instead. Change the first line to...
#!/bin/bash
Then run with...
./virtualenv_activate.sh
...or...
/bin/bash virtualenv_activate.sh
Edit:
If you want the activation of the virtualenv to change the shell that you call the script from, you need to use the "source" or "dot operator". This ensures that the script is run in the current shell (and therefore changes the current environment)...
source virtualenv_activate.sh
...or...
. virtualenv_activate.sh
As a side note, this is why virtualenv always says you need to use "source" to run it's activate script.
source is an builtin shell command in bash, and is not available in sh. If i remember correctly then virtual env does a lot of path and environment variables manipulation. Even running it as bash virtualenv_blah.sh wont work since this will simply create the environment inside the sub-shell.
Try . virtualenv_activate.sh or source virtualenv_activate.sh this basically gets the script to run in your current environment and all the environment variables modified by virtualenv's activate will be available.
HTH.
Edit: Here is a link that might help - http://ss64.com/bash/period.html
On Mac OS X your proposals seems not working.
I have done it this way. I'am not very happy with solution, but share it anyway here and hope, that maybe somebody will suggest the better one:
In activate.sh I have
echo 'source /Users/andi/.virtualenvs/data_science/bin/activate'
I give execution permissions by: chmod +x activate.sh
And I execute this way:
`./activate.sh`
Notice that there are paranthesis in form of ASCII code 96 = ` ( Grave accent )
For me best way work as below.
Create start-my-py-software.sh and pest below code
#!/bin/bash
source "/home/snippetbucket.com/source/AIML-Server-CloudPlatform/bin/activate"
python --version
python /home/snippetbucket.com/source/AIML-Server-CloudPlatform/main.py
Give file permission to run like below.
chmod +x start-my-py-software.sh
Now run like below
.start-my-py-software.sh
and that's it, start my python based server or any other code.
ubuntu #18.0
In my case, Ubuntu 16.04, the methods above didn't worked well or it needs much works.
I just made a link of 'activate' script file and copy it to home folder(or $PATH accessible folder) and renamed it simple one like 'actai'.
Then in a terminal, just call 'source actai'. It worked!