I'm trying to activate a conda environment on a cluster. However, I keep being returned with the shell needing to be initialized. Ruuning conda init bash does nothing and even when closing the shell and trying again I get the same 'conda is not initialized'?
-bash-4.2$ conda activate libraries/
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
-bash-4.2$ conda init bash
no change /usr/local/miniconda3/condabin/conda
no change /usr/local/miniconda3/bin/conda
no change /usr/local/miniconda3/bin/conda-env
no change /usr/local/miniconda3/bin/activate
no change /usr/local/miniconda3/bin/deactivate
no change /usr/local/miniconda3/etc/profile.d/conda.sh
no change /usr/local/miniconda3/etc/fish/conf.d/conda.fish
no change /usr/local/miniconda3/shell/condabin/Conda.psm1
no change /usr/local/miniconda3/shell/condabin/conda-hook.ps1
no change /usr/local/miniconda3/lib/python3.8/site-packages/xontrib/conda.xsh
no change /usr/local/miniconda3/etc/profile.d/conda.csh
no change /nethome/6966225/.bashrc
No action taken.
-bash-4.2$ conda activate libraries/
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
The conda init command only updates the shell initialization files (e.g., .bashrc) to include shell function setup. One still needs to source the relevant initialization file. Typically, this is done by restarting the shell session or running . ~/.bashrc.
Be aware that the shell also needs to be launched in a way that it loads the relevant initialization file. Most user-initiated shell sessions do this automatically, but in some situations one may need to explicitly include a flag for interactive (-i in bash) or login (-l in bash) sessions to ensure loading of .bashrc or .bash_profile files.
Related
I am hoping to run a simple shell script to ease the management around some conda environments. Activating conda environments via conda activate in a linux os works fine in the shell but is problematic within a shell script. Could someone point me into the right direction as to why this is happening?
Example to repeat the issue:
# default conda env
$ conda info | egrep "conda version|active environment"
active environment : base
conda version : 4.6.9
# activate new env to prove that it works
$ conda activate scratch
$ conda info | egrep "conda version|active environment"
active environment : scratch
conda version : 4.6.9
# revert back to my original conda env
$ conda activate base
$ cat shell_script.sh
#!/bin/bash
conda activate scratch
# run shell script - this will produce an error even though it succeeded above
$ ./shell_script.sh
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
I use 'source command' to run the shell script, it works:
source shell_script.sh
The error message is rather helpful - it's telling you that conda is not properly set up from within the subshell that your script is running in. To be able to use conda within a script, you will need to (as the error message says) run conda init bash (or whatever your shell is) first. The behaviour of conda and how it's set up depends on your conda version, but the reason for the version 4.4+ behaviour is that conda is dependent on certain environment variables that are normally set up by the conda shell itself. Most importantly, this changelog entry explains why your conda activate and deactivate commands no longer behave as you expect in versions 4.4 and above.
For more discussion of this, see the official conda issue on GitHub.
Edit: Some more research tells me that the conda init function mentioned in the error message is actually a new v4.6.0 feature that allows a quick environment setup so that you can use conda activate instead of the old source activate. However, the reason why this works is that it adds/changes several environment variables of your current shell and also makes changes to your RC file (e.g.: .bashrc), and RC file changes are never picked up in the current shell - only in newly created shells. (Unless of course you source .bashrc again). In fact, conda init --help says as much:
IMPORTANT: After running conda init, most shells will need to be closed and restarted for changes to take effect
However, you've clearly already run conda init, because you are able to use conda activate interactively. In fact, if you open up your .bashrc, you should be able to see a few lines added by conda teaching your shell where to look for conda commands. The problem with your script, though, lies in the fact that the .bashrc is not sourced by the subshell that runs shell scripts (see this answer for more info). This means that even though your non-login interactive shell sees the conda commands, your non-interactive script subshells won't - no matter how many times you call conda init.
This leads to a conjecture (I don't have conda on Linux myself, so I can't test it) that by running your script like so:
bash -i shell_script.sh
you should see conda activate work correctly. Why? -i is a bash flag that tells the shell you're starting to run in interactive mode, which means it will automatically source your .bashrc. This should be enough to enable you to use conda within your script as if you were using it normally.
Quick solution for bash: prepend the following init script into your Bash scripts.
eval "$(command conda 'shell.bash' 'hook' 2> /dev/null)"
Done.
For other shells, check the init conf of your shell, copy the following content within the shell conf and prepend it into your scripts.
# >>> conda initialize >>>
...
# <<< conda initialize <<<
You can also use
conda init --all --dry-run --verbose
to get the init script you need in your scripts.
Explanation
This is related with the introduction of conda init in conda 4.6.
Quote from conda 4.6 release log
Conda 4.4 allowed “conda activate envname”. The problem was that setting up your shell to use this new feature was not always straightforward. Conda 4.6 adds extensive initialization support so that more shells than ever before can use the new “conda activate” command. For more information, read the output from “conda init –help”
After conda init is introduced in conda 4.6, conda only expose command
conda into the PATH but not all the binaries from "base". And environment switch is unified by conda activate env-name and conda deactivate on all platforms.
But to make these new commands work, you have to do an additional initialization with conda init.
The problem is that your script file is run in a sub-shell, and conda is not initialized in this sub-shell.
References
Conda 4.6 Release
Unix shell initialization
Shell startup scripts
Using conda activate or source activate in shell scripts does not always work and can throw errors like this. An easy work around it to place source ~/miniconda3/etc/profile.d/conda.sh above any conda activate command in the script:
source ~/miniconda3/etc/profile.d/conda.sh # Or path to where your conda is
conda activate some-conda-environment
This is the solution that has worked for me and will also work if sharing scripts. This also gets around having to use conda init as on some clusters I have worked with the system is initialised but conda activate still won't work in a shell script.
if you want to use the shell script to run the other python file in the other conda env, just run the other file via the following command.
os.system('conda run -n <env_name> python <path_to_other_script>')
What is the problem with simply doing something like this in your shell:
source /opt/conda/etc/profile.d/conda.sh
(The conda init is still marked as Experimental, and thus not sure if it is a good idea to use it yet).
I also had the exact same error when trying to activate conda env from C++ or Python file. I solved it by bypassing the conda activate statement and using the absolute path of the specific conda env.
For me, I set up an environment called "testenv" using conda.
I searched all python environments using
whereis python | grep 'miniconda'
It returned a list of python environments. Then I ran my_python_file.py using the following command.
~/miniconda3/envs/testenv/bin/python3.8 my_python_file.py
You can do the same thing on windows too but looking up for python and conda python environments is a bit different.
This answer from Github worked for me (I'm using Ubuntu so it's not for Windows only):
eval "$(conda shell.bash hook)"
conda activate my_env
I just followed a similar solution like the one from hong-xu
So to run a shell command that calls the script with arguments and using a specific conda environment:
from a jupyter cell, goes like this :
p1 = <some-value>
run = f"conda run -n {<env-name>} python {<script-name>.py} \
--parameter_1={p1}"
!{run}
I didn't find any of the above scripts useful. These are fine if you want to run conda in non-interactive mode, but I'd like to run it in interactive mode. If I run:
conda activate my_environment
in a bash script it just runs in the script.
I found that creating an alias in .bashrc is all that is required to change directory to a particular project I'm working on, and set up the correct conda environment for me. So I included in .bashrc:
alias my_environment="cd ~/subdirectory/my_project && conda activate my_environment"
and then:
source ~/.bashrc
Then I can just type at the command line:
my_environment
to change to the correct project and correct environment everytime I want to work on a different project.
This answer is similar to #Lamma answer. This worked for me ->
(1) I defined several variables; the conda activate function, environments directory and environment
conda_activate=~/anaconda3/bin/activate
conda_envs_dir=~/anaconda3/envs
conda_env=<env name>
(2) I source conda activate with the environment
source ${conda_activate} ${conda_envs_dir}/${conda_env}
(3) then you can run your python script
python <path to script.py>
This bypasses the conda init requirement. my .bashrc already was initialized and sourcing the .bashrc file didn't work for me. #Lamma's answer worked for me as well as the above code.
The problem is that when you run the bash script, a new (linux) shell environment is created that was not initialized properly. If your intention is to activate a conda environment, and then run python through the script, you can properly initialize the created shell
environment as discussed in the accepted solution.
If however you want to have the conda environment active after you finish this script, then this will not work because the conda environment has changed on the new shell and you exit that shell when you finish the script. Think of this as running bash, then conda activate... then running exit to exit that bash... More details in How to execute script in the current shell on Linux?:
TL;DR:
Add the line #!/bin/bash as the first line of the script
Type the command source shell_script.sh or . shell_script.sh
Note: . in bash is equivalent to source in bash.
$ conda activate scratch
or
$ source activate scratch
#open terminal or CMD as administrator
$ cd <path Anaconda3 install>\Scripts
$ activate
$ cd ..
$ conda activate scratch
I want to run this code:
#! /bin/sh
env_name="base"
source activate $env_name
in my shell as sh filename.sh, but an error say:
filename.sh: source: not found
with the following line:
filename.sh: Syntax error: Unterminated quoted string
what is the problem? It is good to say that I can run this command in shell manually but when I try to run as shell script this problem gives me the error. any help appreciated.
There is no command named source in POSIX shell. To source a file, use a period, i.e.
. filename.sh
Options
You have three options:
Login Mode - use a shell in login mode that automatically sets up Conda's shell functionality.
Manual initialization - where you manually source the shell-appropriate initialization script to enable conda activate functionality in your script.
conda run - where you let Conda auto-activate the env for you.
1: Login Mode
Typical Conda installations have users run conda init for their primary shell, which adds code to the user's shell-specific resource file (e.g., .bashrc). The current way OP runs the script, and the script's shebang, will never load such a resource and therefore do not define the conda activate function.
Instead, redefine the shebang to use a full shell and run in login mode. For example, this would be the BASH version:
#!/usr/bin/env bash -l
and run with either
. filename.sh
or
bash -l filename.sh
This assumes the user has run conda init bash.
2: Manual Initialization
The conda activate functionality is normally a shell function that Conda creates at the start up of your interactive shells. To use this same functionality in scripts, you can manually run the initialization script. Assuming a bash type shell, this is located under the base Anaconda/Miniconda directory at etc/profile.d/conda.sh. Let's assume you have it at:
/Users/user/miniconda3/etc/profile.d/conda.sh
A working version of your script would then go something like
#!/bin/sh
. /Users/user/miniconda3/etc/profile.d/conda.sh
conda activate base
...
3: Conda Run
You can use conda run (see conda run --help) in the shebang to automatically activate the env and run everything in the script in that context. This assumes that you have properly configured your shell to use conda. For example,
#!/usr/bin/env conda run -n base bash
some_cmd
where some_cmd is a commandline program that is available in your env.
I've searched for an answer and read about different ways to do that in Mac but some of them are not relevant for Mojave or just didn't work for me.
I need to set Environment variable in terminal (bash), run script that creates processes, and I would like those processes to know the value of those environment variables.
How can I do that?
btw - writing export ENV_NAME=ENV_VAL in .bashrc or in .bash_profile didn't work.
Works for me. Have you RTFM? For example, ~/.bashrc is only read by interactive shells, not shell scripts. And ~/.bash_profile is only read by login shells. Again, shell scripts don't usually use the -l flag that would make them login shells. Also, if you put an export VAR=value statement in your ~/.bashrc it won't affect your current interactive shell. You need to start a new shell; e.g., by typing exec bash. Once you do that you should find the env var is defined. And it will be inherited by any process, including a shell script, you launch from that interactive session.
Note that if you run your script via crontab, for example, then you'll need a different means of setting the env var. For example, by using the --init-file flag or the BASH_ENV env var.
Imagine I am in a shell with the working directory set to a directory that contains an .envrc file. When I now open up a new tab in my terminal emulator, a new shell is launched into the same working directory as the original shell.
For some reason, the .envrc file is not loaded in this new shell. If manually exit the working directory and enter it again, the environment variables are then loaded.
Is there a way to make the variables load automatically when the shell is created?
I am running MacOS Mojave (10.14.4) with direnv 2.20.1, using bash as my shell. I am loading the direnv hooks through my .bashrc.
Under macOS Mojave (and probably older versions), saving a .bashrc has no effect. Instead, use a .bash_profile.
I found in my zsh environment I needed to put export in front of each variable in .envrc
VAR=x
doesn't work
Instead, use:
export VAR=x
Is there a way to make the variables load automatically when the shell is created
you can add this to your .bashrc
eval "$(direnv hook bash)"
it activate a hook on at the start of your bash.
If your bashrc is not loaded look at the bash_profile / bashrc to see which calls whom.
you can try to " source ~/.bashrc" to see if its your eval command that is not well written or your bashrc not called.
I'm using MacOS and bash.
I'm using pipenv for virtual environments and I really want to make the look of my virtual shell match the look of my standard shell.
When I'm outside of pipenv, I set my .bash_profile so I could have the time, a green command prompt that shows the full path of my working directory and, if that directory has a git repo, it shows my current branch at the end.
I've searched for the correct file to edit to get the same look for each new pipenv shell, but I'm not having any luck finding the right file to copy my profile preferences to.
I'm wondering how I can set it up with my standard PS1 terminal appearance (colors and current git branch, etc).
pipenv-FAQ
Look at the shell does not show the virtualenv’s name in prompt:
This is intentional. You can do it yourself with either shell plugins, or clever PS1 configuration. If you really want it back, use
And for the "shell plugins"
You need to look not for "pipenv+bash" but for virtualenv+bash prompt solutions, such as:
How do I change the default virtualenv prompt?
It's hard for me to recommend "Which way is the best?"
(I'm using zsh myself)
I've searched for the correct file to edit to get the same look for each new pipenv shell, but I'm not having any luck finding the right file to copy my profile preferences to.
For bash, you can create and modify ~/.bashrc.
The pipenv shell command is a wrapper for virtualenv, which checks for a ~/.bashrc file when it activates a virtual environment for a bash shell. Anything in the ~/.bashrc file gets used in the spawned shell, including prompts, aliases, and other things you customized in your shell.
$ cat ~/.bashrc
cat: /Users/gino/.bashrc: No such file or directory
$ pipenv shell
Launching subshell in virtual environment…
bash-5.0$ . /Users/gino/.venvs/test-oiEjhH94/bin/activate
(test) bash-5.0$ ll
bash: ll: command not found
(test) bash-5.0$ exit
$ vim ~/.bashrc
$ cat ~/.bashrc
PS1='\[\e[1;33m\]\u#\W\$\[\e[0m\] '
alias ll="ls -Flh"
$ pipenv shell
Launching subshell in virtual environment…
gino#test$ . /Users/gino.mempin/.venvs/test-oiEjhH94/bin/activate
(test) gino#test$
(test) gino#test$ type ll
ll is aliased to `ls -Flh'
What you can do is to have a ~/.bash_prompt file where you set all your prompt configs (PS1) and a ~/.bash_aliases file where you set all your aliases. Then, source them both in the ~/.bash_profile (for the standard shell) and in the ~/.bashrc (for the pipenv shell).
~/.bash_prompt
# Format the prompt
# Shows up as:
# machine-name#current-working-directory$
PS1='\[\e[1;33m\]\u#\W\$\[\e[0m\] '
# other prompt configs
~/.bash_aliases
alias ll="ls -FlhpG"
# other aliases
~/.bash_profile
# For "main" shell (i.e. Terminal, VS Code terminal)
. ~/.bash_prompt
. ~/.bash_aliases
~/.bashrc
# For `pipenv shell`
. ~/.bash_prompt
. ~/.bash_aliases
A better solution would be to tell pipenv shell or virtualenv to reuse the configs in the ~/.bash_profile, but I haven't got that successfully to work (Terminal, pipenv, VS Code).
macOS 10.15
bash 5.0 (installed via Homebrew)
pipenv 2020.8.13