I want to use snakemake with fish shell and conda environments in my managed environment (basically I have no root rights and the default shell cannot be changed).
I set up fish as the 'default' shell using this hack inside the .bashrc:
if [ "$REALBASH" != "1" ]; then
case "$-" in
*i*)
export SHELL=/usr/bin/fish
tty > /dev/null && echo "Your ~/.bashrc is switching interactive SHELL to $SHELL"
[ -x $SHELL ] && exec $SHELL "$#"
echo "Apparently $SHELL is not available here. Continuing with bash."
export SHELL=/bin/bash
;;
esac
fi
There is also a command realbash that sets the environment variable REALBASH=1 to bypass this hack.
I managed to get conda to work with fish using this, but it has the disadvantage that within fish the command to activate conda environments is different from bash. In bash, the command is source activate ... and in fish it is conda activate ....
Activating environments works both from bash using source activate ... and from fish using conda activate ....
When I now execute snakemake from fish, I get the following error:
Activating conda environment ...
source: Error encountered while sourcing file “activate”:
source: No such file or directory
If I execute snakemake from bash, the same error occurs.
If I execute snakemake from bash via snakemake --overwrite-shellcmd realbash, I get the same error and end up in the bash shell that was opened by snakemake. Only after typing exit, snakemake completes (but unsuccessfully, of course).
If I execute snakemake from fish via snakemake --overwrite-shellcmd realbash, the same behaviour occurs.
I am confused by the behaviour of --overwrite-shellcmd, is there a way to make this work with my hack?
Otherwise, can I configure snakemake to call conda activate instead of source activate?
Or is there any other solution to this?
Apparently this was a bug in an older version of snakemake. The effects described in the question were produced with snakemake 4.3.1.
Running snakemake from within a conda environment where snakemake 5.17.0 is installed works perfectly fine with the setup as described in the question. No --overwrite-shellcmd or other changes are required.
Related
Im trying to extract current working directory and activate conda environment using bash script. My shell script
app_path=$(dirname $(readlink -f $0))
source /conda/bin/activate $app_path;
This issue im facing is when I run the script as
sh script.sh
it is not activating the conda env
If i run with source script.sh
It is not able to recognise readlink command. How to get both of them working together?
If I run
xterm -hold
and within the new terminal I type
conda activate my_environment
the conda environment "my_environment" is indeed activated.
However, when passing this command using the -e flag, it does not work:
xterm -hold -e "conda activate my_environment"
It instead returns the following error message:
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
So, how can this be done using xterm? Or should I use another type of external terminal?
Background
The conda activate command is a shell function that gets defined during initialization of the shell. conda init adds code to the initialization file (e.g., .bash_profile) to run the scripts that define the conda activate shell function.
Solutions
Possible fix: xterm options
When using the -c argument with xterm it no longer runs the initialization script. Hence, conda activate never gets defined. For bash there is the -l that tells it to run the init files. I expected xterm's -ls argument to trigger similar behavior, but it didn't work for me. Perhaps someone more familiar can point you to the correct flag.
Manually Run Conda Script
Otherwise, you just run the Conda script yourself (assuming it's the bash version). Either of these will work:
xterm -hold -e ". /path/to/miniconda3/etc/profile.d/conda.sh && conda activate my_environment && which python"
or
xterm -hold -e "$(conda shell.bash hook) && conda activate my_environment && which python"
The which python is only included to show that you're getting the env activated.
Conda Run
Another option is conda run, which automates executing commands under an environment. The follow is equivalent to what I did in the last section, but without having to know what shell I am running in:
xterm -hold -e "conda run -n my_environment which python"
Just be aware that this functionality is still a work-in-progress. Personally, I've found it quite useful for running simple scripts in specific envs and haven't run into problems.
I am using conda 4.6.8 to test a python package in a conda env on Travis CI. I want to replace my old source activate ENVNAME line with the new conda activate ENVNAME command in my Travis CI configuration. If I run this on Travis:
>>> conda update -n base conda
>>> conda init
no change /home/travis/miniconda/condabin/conda
no change /home/travis/miniconda/bin/conda
no change /home/travis/miniconda/bin/conda-env
no change /home/travis/miniconda/bin/activate
no change /home/travis/miniconda/bin/deactivate
no change /home/travis/miniconda/etc/profile.d/conda.sh
no change /home/travis/miniconda/etc/fish/conf.d/conda.fish
no change /home/travis/miniconda/shell/condabin/Conda.psm1
no change /home/travis/miniconda/shell/condabin/conda-hook.ps1
no change /home/travis/miniconda/lib/python3.7/site-packages/xonsh/conda.xsh
no change /home/travis/miniconda/etc/profile.d/conda.csh
modified /home/travis/.bashrc
==> For changes to take effect, close and re-open your current shell. <==
How can I "close and re-open" my shell on Travis? Because otherwise I cannot activate my conda environment:
>>> conda create -n TEST package_names
>>> conda activate TEST
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
The command "conda activate TEST" failed and exited with 1 during .
Your build has been stopped.
Not sure it is currently supported as the official doc still uses source in travis.yml.
What does conda init do?
This new command should harmonize the way users setup their shells to be able to call conda activate.
Actually, if you run conda init --dry-run --verbose you will see that it tries to source conda.sh from your ~/.bashrc (assuming you're running Bash, from info mentioned in your question).
And conda.sh will define a conda() function that will catch a few commands among which activate and deactivate and dispatch to $CONDA_EXE:
conda() {
if [ "$#" -lt 1 ]; then
"$CONDA_EXE"
else
\local cmd="$1"
shift
case "$cmd" in
activate|deactivate)
__conda_activate "$cmd" "$#"
;;
install|update|upgrade|remove|uninstall)
"$CONDA_EXE" "$cmd" "$#" && __conda_reactivate
;;
*) "$CONDA_EXE" "$cmd" "$#" ;;
esac
fi
}
So unless this function is defined in your local shell, you won't be able to call conda activate.
Hint on a solution? (not tested for Travis CI)
The only hint I can suggest is to try source $(conda info --root)/etc/profile.d/conda.sh and then conda activate. This should do roughly the same as conda init assuming you are using Bourne shell derivatives.
For csh there is $(conda info --root)/etc/profile.d/conda.csh, and for fish there is $(conda info --root)/etc/fish/conf.d/conda.fish
Note: although not tested for Travis CI, this solution works for me from bash. Of course, the conda executable should be found in PATH for conda info --root to work properly.
I'm trying to activate my pyenv-virtualenv environment through very simple bash script like this.
set -e
pyenv activate myenv
But I can't activate my env with below error:
Failed to activate virtualenv.
Perhaps pyenv-virtualenv has not been loaded into your shell properly.
Please restart current shell and try again.
I can activate it in command line, so I think it's due to my shell environment.
I use fish shell, but it's launched after $PATH settings read in bash shell.
I show you my settings.
.bash_profile:
export LANG="ja_JP.UTF-8"
export XDG_CONFIG_HOME="$HOME/.config"
export PATH="/sbin"
export PATH="/usr/sbin:$PATH"
export PATH="/bin:$PATH"
export PATH="/usr/bin:$PATH"
export PATH="/usr/local/bin:$PATH"
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
exec fish
And my config.fish:
status --is-interactive; and source (pyenv init -|psub)
status --is-interactive; and source (pyenv virtualenv-init -|psub)
What's wrong?
I'm using:
OS: OS X 10.13.5
fish: v2.7.1
pyenv: v1.2.5
pyenv-virtualenv: v1.1.3
I believe the cause is that you are initializing PyEnv in the Fish shell environment but then calling PyEnv in the Bash shell environment (via your Bash script). To solve this problem, you could try either of the following potential solutions:
Initialize PyEnv in your Bash environment (in addition to your Fish environment)
Write your script in Fish instead of Bash
Since PyEnv seems so have better support for Bash than Fish, let's focus on option #1. Step 3 of the PyEnv installation guide suggests the following command for adding PyEnv initialization to your ~/.bash_profile:
$ echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bash_profile
After running that command, I suspect your script will function as you expect. If it does not, you could try adding the same PyEnv initialization snippet (that you just added to your ~/.bash_profile) to the top of your Bash script.
I want to use rvm (or rbenv/chruby for that matter) to select different ruby versions from within my Jenkins jobs.
By default, Jenkins will use /bin/sh, which on Ubuntu, is dash.
For this to change, I can add
#!/bin/bash -l
To the top of every single shell execute function everywhere. Seeing as that's a lot of annoying work, I'd like to be able to set that somewhere central.
Using the "Shell executable" configuration setting, I can get it to run bash, adding parameters like '-l' however will fail with
"/bin/bash -l" -xe /tmp/hudson5660076222778817826.sh FATAL:
command execution failed java.io.IOException: Cannot run program
"/bin/bash -l" (in directory
"/home/jenkins/jobs/workspace/rvm-test"): error=2, No such file or
directory
I tried using the rvm plugin for jenkins, but that doesn't even install on the current release version.
Any ideas? :)
You could work around by creating a wrapper around bash:
#!/bin/sh
# for ex.: /usr/local/bin/login-bash
exec /bin/bash -l "$#"
If you want to use the default ruby just use the rvm-shell, which comes with rvm.
Login as the jenkins user and type:
$ which rvm-shell
/home/jenkins/.rvm/bin/rvm-shell
to get the path of the rvm-shell.
Use this path for the "Shell executable" option.